Max tokens explained: input, output, and total limits

When you search for max tokens explained, you’re usually trying to solve one of three problems: your request got rejected with a 400 error, your completion got cut off mid-sentence, or your bill surprised you. The parameter name is deceptively simple — max_tokens in OpenAI’s API, max_output_tokens in Anthropic’s, max_new_tokens in some open-source servers — but the behavior differs across providers and models in ways that break production code. This guide walks through the mechanics, the gotchas, and the patterns that keep your inference pipeline reliable.

What max tokens actually controls

max_tokens (or its provider-specific equivalent) sets a hard ceiling on the number of tokens the model may generate. It does not limit the prompt. The prompt consumes space from the model’s context window — typically 4K, 8K, 32K, 128K, or 1M+ tokens depending on the model — and the generation must fit in whatever remains.

context_window = prompt_tokens + completion_tokens
completion_tokens ≤ max_tokens

If prompt_tokens + max_tokens > context_window, the request fails before generation starts. Most APIs return a 400 with a message like “This model’s maximum context length is 8192 tokens, but you requested 9000 tokens.”

Common pitfall: Developers set max_tokens: 4096 on an 8K model, send a 6K prompt, and wonder why they get an error. The math is 6000 + 4096 > 8192. The fix is either reducing max_tokens or truncating the prompt.

Input vs. output vs. total: the three budgets

The total budget is the hard architectural constraint. The input and output budgets are policy knobs you turn per request.

Provider differences matter

• OpenAI: max_tokens defaults to inf (context window minus prompt). You must set it explicitly for cost predictability.
• Anthropic: max_tokens is required. No default. Forces you to think about output budget upfront.
• Google (Gemini): maxOutputTokens optional, defaults vary by model. Vertex AI and AI Studio differ slightly.
• Open-source (vLLM, TGI, Ollama): max_new_tokens or max_tokens. Some servers ignore values larger than the model’s trained max and silently clamp.

Rule of thumb: Always set an explicit output budget. Never rely on defaults. It makes your token spend predictable and your finish_reason handling deterministic.

Calculating safe limits programmatically

Don’t hardcode context windows. Models get updated, new variants launch, and your code breaks silently. Fetch the limit from the model metadata or maintain a local registry.

# Minimal registry pattern — extend as you adopt new models
MODEL_CONTEXT_WINDOWS = {
    "gpt-4o": 128_000,
    "gpt-4o-mini": 128_000,
    "gpt-4-turbo": 128_000,
    "gpt-3.5-turbo": 16_385,
    "claude-3-5-sonnet-20241022": 200_000,
    "claude-3-haiku-20240307": 200_000,
    "gemini-1.5-pro": 2_000_000,
    "gemini-1.5-flash": 1_000_000,
    "llama-3.1-70b": 131_072,
    "llama-3.1-8b": 131_072,
}

def max_safe_output_tokens(model: str, prompt_tokens: int, reserve: int = 256) -> int:
    """
    Return the largest max_tokens you can request without hitting the context limit.
    `reserve` leaves headroom for special tokens, function call overhead, etc.
    """
    window = MODEL_CONTEXT_WINDOWS.get(model)
    if window is None:
        raise ValueError(f"Unknown model: {model}")
    available = window - prompt_tokens - reserve
    return max(0, available)

Use a tokenizer to count prompt tokens before sending the request. tiktoken for OpenAI models, anthropic-tokenizer for Claude, or the provider’s count_tokens endpoint if available.

import tiktoken

def count_prompt_tokens(messages: list[dict], model: str) -> int:
    encoding = tiktoken.encoding_for_model(model)
    # Rough but practical: count content + role overhead
    total = 0
    for msg in messages:
        total += 4  # message framing overhead
        for key, value in msg.items():
            if isinstance(value, str):
                total += len(encoding.encode(value))
    total += 2  # assistant priming
    return total

Handling finish_reason: "length" in production

When the model hits max_tokens, the response includes finish_reason: "length" (OpenAI) or stop_reason: "max_tokens" (Anthropic). The output is truncated — often mid-sentence, mid-code-block, or mid-JSON. Your code must detect this and decide: retry with a larger budget, return partial to the user, or fail explicitly.

from openai import OpenAI
from openai.types.chat import ChatCompletion

client = OpenAI()

def complete_with_budget(
    messages: list[dict],
    model: str,
    max_tokens: int,
    max_retries: int = 1,
) -> ChatCompletion:
    for attempt in range(max_retries + 1):
        resp = client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=0.2,
        )
        choice = resp.choices[0]
        if choice.finish_reason == "length":
            if attempt < max_retries:
                # Increase budget by 50% and retry once
                max_tokens = int(max_tokens * 1.5)
                continue
            # Last attempt: return partial but flag it
            choice.message.content += "\n\n[TRUNCATED: increase max_tokens]"
        return resp
    raise RuntimeError("Unreachable")

Tradeoff: Retrying with a larger budget costs more and adds latency. For user-facing chat, returning partial with a “continue” button is often better. For structured extraction (JSON, code), truncation corrupts the output — fail fast and ask the caller to increase the budget.

Streaming doesn’t change the math

Streaming (stream: true) delivers tokens as they’re generated, but the max_tokens limit still applies. The final chunk carries finish_reason: "length" if the budget was exhausted. You still need to handle truncation.

def stream_with_limit(messages: list[dict], model: str, max_tokens: int):
    stream = client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=max_tokens,
        stream=True,
    )
    collected = []
    finish_reason = None
    for chunk in stream:
        delta = chunk.choices[0].delta
        if delta.content:
            collected.append(delta.content)
            yield delta.content
        if chunk.choices[0].finish_reason:
            finish_reason = chunk.choices[0].finish_reason
    if finish_reason == "length":
        yield "\n\n[TRUNCATED]"

Pitfall: Some providers (notably older Anthropic API versions) don’t emit finish_reason on the final streaming chunk. Check the current docs for your provider version.

Stop sequences: the other output boundary

stop sequences (OpenAI) / stop_sequences (Anthropic) terminate generation before max_tokens is reached. They’re evaluated per-token: as soon as the generated text ends with any stop string, generation halts and finish_reason: "stop".

# Stop at the next user turn in a chat template
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    max_tokens=500,
    stop=["\nUser:", "\nHuman:", "<|im_end|>"],
)

Interaction with max_tokens: Whichever boundary hits first wins. If you set max_tokens: 100 and a stop sequence would appear at token 120, you get 100 tokens and finish_reason: "length". If the stop sequence appears at token 80, you get 80 tokens and finish_reason: "stop".

Use case: Stop sequences are cleaner than post-hoc truncation for structured formats. Generating JSON? Stop at }\n or }]. Generating code? Stop at \n```\n. But don’t rely on them exclusively — the model might never emit the stop string. Always keep max_tokens as a safety net.

Prompt truncation strategies

When the prompt exceeds the available context, you must truncate before the request. Three common strategies:

1. Drop oldest messages (sliding window)

def truncate_messages_sliding(
    messages: list[dict],
    model: str,
    max_tokens: int,
    system_prompt: str | None = None,
) -> list[dict]:
    """Keep system prompt + most recent messages that fit."""
    encoding = tiktoken.encoding_for_model(model)
    window = MODEL_CONTEXT_WINDOWS[model]
    budget = window - max_tokens - 256  # reserve
    
    # Always keep system prompt if present
    system_msgs = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]
    
    kept = []
    total = sum(len(encoding.encode(m["content"])) for m in system_msgs)
    kept.extend(system_msgs)
    
    # Add newest first
    for msg in reversed(other_msgs):
        cost = len(encoding.encode(msg["content"])) + 4
        if total + cost > budget:
            break
        kept.insert(len(system_msgs), msg)  # preserve order after system
        total += cost
    
    return kept

2. Summarize older history

async def summarize_and_truncate(
    messages: list[dict],
    model: str,
    max_tokens: int,
    keep_recent: int = 4,
) -> list[dict]:
    if len(messages) <= keep_recent + 1:  # +1 for system
        return messages
    
    to_summarize = messages[1:-keep_recent]  # skip system, keep recent
    summary_prompt = [
        {"role": "system", "content": "Summarize the following conversation in 3-4 sentences. Preserve key facts, decisions, and open questions."},
        *to_summarize,
    ]
    summary_resp = await client.chat.completions.create(
        model=model,
        messages=summary_prompt,
        max_tokens=300,
        temperature=0,
    )
    summary = summary_resp.choices[0].message.content
    
    return [
        messages[0],  # system
        {"role": "system", "content": f"Previous conversation summary: {summary}"},
        *messages[-keep_recent:],
    ]

3. RAG-style retrieval (for very long contexts)

Instead of stuffing everything into the prompt, retrieve relevant chunks at query time. This is a separate architecture but worth mentioning: the effective context becomes the retrieved snippets plus the query, not the full history.

Tradeoff: Sliding window is simple but loses context. Summarization preserves gist but adds latency and cost. Retrieval scales best but requires embedding infrastructure. Choose based on your latency budget and how much history actually matters for the task.

Cost control: max_tokens as a spend ceiling

max_tokens is your primary cost control lever. Output tokens cost 2-10x more than input tokens on most providers. A runaway generation (e.g., model stuck in a loop) can burn thousands of dollars in minutes if you don’t cap it.

# Estimate cost before sending — useful for guardrails
OUTPUT_PRICE_PER_1K = {
    "gpt-4o": 0.01,
    "gpt-4o-mini": 0.0006,
    "claude-3-5-sonnet-20241022": 0.015,
    "claude-3-haiku-20240307": 0.00125,
}

def estimate_max_cost(model: str, max_tokens: int) -> float:
    price = OUTPUT_PRICE_PER_1K.get(model, 0.01)  # conservative default
    return (max_tokens / 1000) * price

# In your request wrapper:
MAX_ALLOWED_COST_USD = 0.50  # per request ceiling

def guarded_complete(messages: list[dict], model: str, max_tokens: int):
    est_cost = estimate_max_cost(model, max_tokens)
    if est_cost > MAX_ALLOWED_COST_USD:
        raise ValueError(f"Request would cost ~${est_cost:.4f}, limit is ${MAX_ALLOWED_COST_USD}")
    return client.chat.completions.create(model=model, messages=messages, max_tokens=max_tokens)

Production tip: Log actual usage.completion_tokens on every response. Alert on p99 approaching your max_tokens setting — it means you’re truncating useful output or your prompt is too verbose.

Model-specific quirks to know

o1 reasoning tokens: This is the most expensive gotcha in 2024. max_completion_tokens on o1 models covers both the visible answer and the hidden chain-of-thought. A 500-token answer might consume 2,000 completion tokens. Set budgets accordingly and monitor usage.completion_tokens_details.reasoning_tokens if available.

Putting it together: a request wrapper

from dataclasses import dataclass
from typing import Optional
import tiktoken
from openai import OpenAI

@dataclass
class CompletionConfig:
    model: str
    max_tokens: int
    temperature: float = 0.2
    stop: Optional[list[str]] = None
    max_cost_usd: float = 0.50

class SafeCompleter:
    def __init__(self, client: OpenAI):
        self.client = client
        self.encoding_cache = {}
    
    def _encoding(self, model: str):
        if model not in self.encoding_cache:
            self.encoding_cache[model] = tiktoken.encoding_for_model(model)
        return self.encoding_cache[model]
    
    def _count_tokens(self, messages: list[dict], model: str) -> int:
        enc = self._encoding(model)
        total = 0
        for m in messages:
            total += 4
            for k, v in m.items():
                if isinstance(v, str):
                    total += len(enc.encode(v))
        total += 2
        return total
    
    def complete(self, messages: list[dict], config: CompletionConfig):
        # 1. Validate context fit
        prompt_tokens = self._count_tokens(messages, config.model)
        window = MODEL_CONTEXT_WINDOWS.get(config.model)
        if window is None:
            raise ValueError(f"Unknown model: {config.model}")
        
        if prompt_tokens + config.max_tokens > window - 256:
            raise ValueError(
                f"Prompt ({prompt_tokens} tokens) + max_tokens ({config.max_tokens}) "
                f"exceeds context window ({window})"
            )
        
        # 2. Cost guardrail
        est_cost = estimate_max_cost(config.model, config.max_tokens)
        if est_cost > config.max_cost_usd:
            raise ValueError(f"Estimated cost ${est_cost:.4f} exceeds limit ${config.max_cost_usd}")
        
        # 3. Execute
        resp = self.client.chat.completions.create(
            model=config.model,
            messages=messages,
            max_tokens=config.max_tokens,
            temperature=config.temperature,
            stop=config.stop,
        )
        
        # 4. Handle truncation
        choice = resp.choices[0]
        if choice.finish_reason == "length":
            # Log for monitoring
            print(f"WARNING: Truncated at {config.max_tokens} tokens", extra={
                "model": config.model,
                "prompt_tokens": prompt_tokens,
                "finish_reason": "length",
            })
        
        return resp

Checklist before you ship

• Explicit max_tokens on every request — no defaults.
• Prompt token counting before send — fail fast if over budget.
• finish_reason: "length" handling — retry, truncate gracefully, or error.
• Stop sequences for structured output — but keep max_tokens as backstop.
• Cost ceiling per request — alert on actual spend vs estimate.
• Model registry with context windows — update when providers launch new models.
• o1 reasoning token budget — multiply visible output budget by 3-4x.
• Streaming truncation — final chunk carries the finish reason, handle it.

Summary

max_tokens explained simply: it’s the output budget. The input consumes the context window; the output must fit in what’s left. Set it explicitly, count prompt tokens first, handle finish_reason: "length" deliberately, and put a dollar ceiling on every request. The providers differ in parameter names, defaults, and quirks (especially o1 reasoning tokens), but the math is universal: prompt + completion ≤ context_window. Build your wrapper once, reuse it everywhere, and stop debugging 400 errors at 2 AM.