Common max_tokens mistakes that truncate JSON output

If you’ve ever parsed a model response only to find a dangling comma or an unclosed brace, you’ve hit a max_tokens json truncation mistake. The symptom looks like a parser error, but the root cause is almost always a token budget miscalculation. Below are the seven most common ways engineers undershoot max_tokens when generating structured output, and how to fix each one.

1. Setting max_tokens to the exact token count of your expected JSON

You counted the characters in your ideal output, divided by four, and set max_tokens to that number. The model returns valid JSON — until it doesn’t. A single extra adjective in a description field, a longer UUID, or an additional array element pushes the completion past the limit and the response gets cut mid-token.

The fix: add a 20–30% buffer on top of your estimated output tokens. For a 500-token JSON payload, request 650–700. If you’re using a provider that charges per output token, the extra cost is negligible compared to the retry loop you’ll otherwise build.

# Bad: exact budget
max_tokens = estimated_json_tokens

# Good: buffer for variance
max_tokens = int(estimated_json_tokens * 1.3)

2. Ignoring reasoning tokens on o1-class models

Models like o1, o3-mini, and DeepSeek-R1 emit reasoning tokens before the final answer. Those tokens count against max_completion_tokens (the OpenAI parameter that replaced max_tokens for reasoning models), but they never appear in the response body. If you set max_completion_tokens to your JSON budget, the reasoning trace consumes most of it and the actual output truncates.

Set max_completion_tokens to: reasoning_budget + json_budget + buffer. OpenAI suggests 10k–20k for complex reasoning tasks. If you cap at 4k because “JSON is small,” you’ll get a thoughtful essay and zero valid JSON.

{
  "model": "o3-mini",
  "max_completion_tokens": 15000,
  "response_format": { "type": "json_object" }
}

3. Forgetting JSON syntax overhead

A schema with 20 fields of 10-character strings looks like 200 tokens of data. But braces, quotes, colons, commas, and whitespace add 30–50% overhead. Nested objects and arrays multiply it. A 1,000-token payload estimate becomes 1,400+ tokens on the wire.

Count tokens on the serialized JSON, not the data values. Use the provider’s tokenizer (tiktoken for OpenAI, Anthropic’s token counter) on a realistic sample payload before you hardcode a limit.

import tiktoken
import json

enc = tiktoken.encoding_for_model("gpt-4o")
sample = {"items": [{"id": "uuid", "name": "Product Name", "tags": ["tag1", "tag2"]} for _ in range(50)]}
serialized = json.dumps(sample, separators=(",", ":"))
token_count = len(enc.encode(serialized))
# token_count is your real baseline

4. Not using stop sequences to guarantee closure

Even with a generous max_tokens, the model may stop mid-array because it “feels” done. A stop sequence on the closing brace of your root object forces the model to complete the structure or hit the hard limit trying.

{
  "model": "gpt-4o",
  "max_tokens": 2000,
  "stop": ["}"],
  "response_format": { "type": "json_object" }
}

Caveat: this only works if your root is a single object. If your schema allows multiple top-level values, stop on the sequence that unambiguously terminates your format (e.g., ]} for an array of objects).

5. Streaming without a reassembly buffer

Streaming parsers like json-stream or simdjson can consume partial JSON, but only if you feed them complete chunks. If your HTTP client yields chunks split mid-token (common with SSE), the parser sees invalid UTF-8 or truncated strings and errors out.

Accumulate chunks into a buffer, decode as UTF-8 with errors="replace", then feed the buffer to the streaming parser. Discard the buffer only after the parser emits a complete event.

import json
from json_stream import loads

buffer = ""
for chunk in response.iter_lines():
    buffer += chunk.decode("utf-8", errors="replace")
    try:
        for event in loads(buffer):
            yield event
        buffer = ""  # parser consumed everything
    except json.JSONDecodeError:
        continue  # wait for more data

6. Assuming token counts are consistent across providers

Anthropic, OpenAI, Google, and open-weight models tokenize differently. The same JSON string can be 1,200 tokens on GPT-4o, 1,450 on Claude 3.5 Sonnet, and 1,800 on Llama-3.1-70B. If you route requests to multiple providers (as n4n.ai does automatically when a provider degrades), a single max_tokens value will truncate on the highest-token-count provider.

Either set max_tokens per-provider based on that provider’s tokenizer, or set it to the maximum across your provider set plus buffer. The latter wastes tokens on efficient providers but guarantees completion everywhere.

PROVIDER_TOKEN_MULTIPLIERS = {
    "openai": 1.0,
    "anthropic": 1.2,
    "google": 1.15,
    "meta": 1.5,
}

def max_tokens_for_provider(base_tokens: int, provider: str) -> int:
    return int(base_tokens * PROVIDER_TOKEN_MULTIPLIERS[provider] * 1.3)

7. Validating only after the full response arrives

Waiting for the entire response to validate JSON means you discover truncation after burning the full token budget and latency. Incremental validation — checking bracket balance, quote pairing, and required keys as tokens arrive — lets you abort early, retry with a higher limit, or fall back to a smaller schema.

A lightweight state machine tracking {, [, ", and : depth catches 90% of truncation cases before the parser does.

class JSONTruncationDetector:
    def __init__(self, required_keys: set[str]):
        self.depth = 0
        self.in_string = False
        self.escape = False
        self.seen_keys = set()
        self.required_keys = required_keys
        self.current_key = ""
        self.reading_key = False

    def feed(self, char: str) -> bool:
        if self.escape:
            self.escape = False
            return True
        if char == "\\":
            self.escape = True
            return True
        if char == '"' and not self.escape:
            self.in_string = not self.in_string
            if not self.in_string and self.reading_key:
                self.seen_keys.add(self.current_key)
                self.current_key = ""
                self.reading_key = False
            elif self.in_string and self.depth > 0:
                self.reading_key = True
            return True
        if self.in_string:
            if self.reading_key:
                self.current_key += char
            return True
        if char in "{[":
            self.depth += 1
        elif char in "}]":
            self.depth -= 1
            if self.depth < 0:
                return False  # invalid structure
        return True

    def is_complete(self) -> bool:
        return self.depth == 0 and not self.in_string and self.required_keys.issubset(self.seen_keys)

Summary

The pattern across all seven: treat max_tokens as a guarantee you must engineer for, not a limit you hope suffices. Budget for the worst-case provider, the worst-case reasoning trace, and the worst-case serialization overhead. Then add the buffer.