Why your LLM response cuts off mid-sentence

Your LLM response cuts off mid-sentence for one of three reasons: you hit max_tokens, a stop sequence fired, or the provider enforced a hidden limit. The fix depends entirely on which one actually happened. Most engineers waste hours checking the wrong thing because the symptoms look identical in the logs.

The three culprits look the same in production

You send a request. The model generates. The response ends abruptly — no period, no closing brace, just silence. Your logging shows finish_reason: "length" or finish_reason: "stop" or sometimes nothing useful at all.

{
  "choices": [{
    "message": {
      "content": "Here is the JSON you requested: {\n  \"users\": [\n    {\"id\": 1, \"name\": \"Alice\"},\n    {\"id\": 2, \"name\": \"Bob\"},\n    {\"id": 3, \"name",
    "role": "assistant"
  },
  "finish_reason": "length"
}

That finish_reason: "length" tells you the model hit a token budget. But which budget? Yours? The provider’s? The model’s context window? The answer changes your fix completely.

Max tokens: the budget you set (and the one you didn’t)

Every request carries a max_tokens (or max_completion_tokens in newer OpenAI APIs) parameter. If you omit it, the provider applies a default — often 1024 or 2048 tokens, sometimes less. That default is the most common cause of mysterious truncation.

# Explicit is better than implicit
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    max_completion_tokens=4096,  # Set it. Always.
)

But there’s a second budget: the model’s output limit, which is distinct from its context window. GPT-4o supports 128k context but caps output at 16k tokens. Claude 3.5 Sonnet: 200k context, 8k output. If your prompt consumes 120k tokens on a 128k model, you have 8k left for completion — but the model may still refuse to generate more than its output cap.

# Calculate what you actually have available
def max_output_tokens(model: str, prompt_tokens: int) -> int:
    limits = {
        "gpt-4o": {"context": 128_000, "output": 16_384},
        "claude-3-5-sonnet-20241022": {"context": 200_000, "output": 8_192},
        "gemini-1.5-pro": {"context": 2_000_000, "output": 8_192},
    }
    model_limit = limits.get(model, {"context": 4096, "output": 2048})
    available = model_limit["context"] - prompt_tokens
    return min(model_limit["output"], available)

Tradeoff: Setting max_tokens too high wastes money on generations you’ll truncate anyway. Setting it too low cuts off valid responses. The right number is min(your_budget, model_output_cap, context_remaining).

Stop sequences: the silent assassins

Stop sequences terminate generation immediately when matched. They don’t complete the token — they cut mid-token if the sequence spans token boundaries. This produces responses that end mid-word, mid-string, mid-anything.

# This stops at "###" but also at "### " or "###\n"
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    stop=["###", "```", "<|im_end|>"],
    max_completion_tokens=4096,
)

Common pitfalls:

1. Inherited stop sequences: Some providers inject stop sequences you didn’t request. Anthropic’s API adds </stop_sequence> for certain model versions. Azure OpenAI may append deployment-specific stops.
2. Tokenization mismatches: Your stop string "\n\n" might tokenize as two tokens. The model generates the first newline, checks for stop, doesn’t match, generates the second — then stops. You lose the trailing newline.
3. Case sensitivity: Stop matching is exact byte/token matching. "STOP" won’t catch "stop".

# Debug: log what the provider actually received
import httpx

class DebugTransport(httpx.BaseTransport):
    def handle_request(self, request):
        print(f"REQUEST: {request.method} {request.url}")
        print(f"BODY: {request.content.decode()}")
        return httpx.Response(200, json={})

# Use with httpx.Client(transport=DebugTransport()) to see raw requests

Tradeoff: Stop sequences save tokens and enforce format. But they’re brittle. Prefer structured output (JSON mode, function calling) over stop-sequence parsing when the schema matters.

Provider limits: the budgets you can’t control

Even with max_tokens=100000, providers enforce hard caps. These vary by model, tier, and sometimes time of day.

Some providers also enforce rate limits that manifest as truncation. You request 4k tokens. The provider allows 1k/minute. The response cuts at ~1k with finish_reason: "length" — but your max_tokens was 4k. The logs lie.

# Detect provider-side truncation vs your limit
def diagnose_truncation(response, requested_max: int) -> str:
    finish = response.choices[0].finish_reason
    usage = response.usage
    
    if finish == "length":
        if usage.completion_tokens >= requested_max:
            return "YOUR max_tokens limit"
        elif usage.completion_tokens >= 16384:  # Known GPT-4o cap
            return "PROVIDER output cap (16k)"
        elif usage.completion_tokens >= 8192:   # Known Claude/Gemini cap
            return "PROVIDER output cap (8k)"
        else:
            return f"UNKNOWN limit at {usage.completion_tokens} tokens"
    elif finish == "stop":
        return f"Stop sequence matched: {response.choices[0].stop_reason}"
    return finish

Tradeoff: You can’t fix provider caps. You can only design around them — chunking, continuation prompts, or switching models.

Streaming changes the failure mode

With streaming, truncation doesn’t return finish_reason: "length" in a final chunk. The stream just ends. No final chunk, no done: true, no error. Your parser sits waiting for data: [DONE] that never arrives.

async def stream_with_timeout(client, messages, max_tokens, timeout=30):
    """Stream with explicit timeout and truncation detection."""
    stream = await client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        max_completion_tokens=max_tokens,
        stream=True,
        stream_options={"include_usage": True},  # Critical: gets usage in final chunk
    )
    
    collected = []
    finish_reason = None
    usage = None
    
    try:
        async for chunk in asyncio.wait_for(stream, timeout=timeout):
            if chunk.choices:
                delta = chunk.choices[0].delta
                if delta.content:
                    collected.append(delta.content)
                if chunk.choices[0].finish_reason:
                    finish_reason = chunk.choices[0].finish_reason
            if chunk.usage:
                usage = chunk.usage
    except asyncio.TimeoutError:
        return "".join(collected), "timeout", usage
    
    return "".join(collected), finish_reason, usage

The stream_options={"include_usage": true} parameter (OpenAI) or equivalent is your only signal that the stream completed normally vs abruptly. Without it, you cannot distinguish “model finished naturally” from “provider cut me off.”

How to debug in production

Add a middleware that logs the actual request sent and the actual response received — not what your SDK claims it sent.

# Middleware pattern for any OpenAI-compatible client
class TruncationLogger:
    def __init__(self, client):
        self.client = client
    
    async def chat_completions_create(self, **kwargs):
        request_id = uuid.uuid4().hex[:8]
        requested_max = kwargs.get("max_completion_tokens") or kwargs.get("max_tokens")
        
        # Log request
        logger.info(f"[{request_id}] REQUEST max_tokens={requested_max} model={kwargs.get('model')}")
        
        try:
            response = await self.client.chat.completions.create(**kwargs)
            
            # Log response
            usage = response.usage
            finish = response.choices[0].finish_reason
            logger.info(
                f"[{request_id}] RESPONSE finish_reason={finish} "
                f"prompt_tokens={usage.prompt_tokens} "
                f"completion_tokens={usage.completion_tokens} "
                f"total_tokens={usage.total_tokens}"
            )
            
            # Flag anomalies
            if finish == "length" and requested_max:
                if usage.completion_tokens < requested_max * 0.9:
                    logger.warning(
                        f"[{request_id}] TRUNCATED EARLY: got {usage.completion_tokens} "
                        f"of {requested_max} requested. Provider limit likely."
                    )
            
            return response
            
        except Exception as e:
            logger.error(f"[{request_id}] ERROR: {e}")
            raise

This catches the “provider limit lower than your limit” case automatically. You’ll see warnings like TRUNCATED EARLY: got 8192 of 16384 requested — that’s your signal to switch models or chunk.

Continuation strategies that actually work

When you must exceed output caps, don’t just re-prompt. The model loses context and hallucinates continuations.

async def generate_long(client, prompt: str, target_tokens: int, model: str) -> str:
    """Generate beyond model output cap via controlled continuation."""
    chunks = []
    remaining = target_tokens
    conversation = [{"role": "user", "content": prompt}]
    
    while remaining > 0:
        # Request min(remaining, model_cap) but leave room for prompt
        chunk_max = min(remaining, 8000)  # Conservative
        
        response = await client.chat.completions.create(
            model=model,
            messages=conversation,
            max_completion_tokens=chunk_max,
        )
        
        content = response.choices[0].message.content
        chunks.append(content)
        remaining -= response.usage.completion_tokens
        
        finish = response.choices[0].finish_reason
        if finish == "stop" or finish is None:
            break  # Model finished naturally
        
        if finish == "length":
            # Continue from exact cutoff point
            conversation.append({"role": "assistant", "content": content})
            conversation.append({
                "role": "user", 
                "content": "Continue exactly where you left off. Do not repeat. Do not summarize."
            })
        else:
            break
    
    return "".join(chunks)

Key details:

• Pass the entire previous response as assistant message, not a summary
• Explicit instruction: “Continue exactly where you left off”
• Track token usage per chunk, not just character count
• Stop on finish_reason: "stop" — the model decided it was done

Tradeoff: Continuation adds latency (sequential requests) and cost (re-processing context). But it’s the only reliable way past hard output caps.

The decisive takeaway

Your LLM response cuts off mid-sentence because of a token budget. The budget is either yours (max_tokens), the model’s (output cap), or a stop sequence you didn’t know existed.

Always set max_completion_tokens explicitly. Never rely on defaults. Log finish_reason and usage.completion_tokens on every request. The ratio of completion_tokens / max_tokens tells you which budget won: ~1.0 means your limit, ~0.5 means provider limit, ~0.0 means stop sequence.

If you need more tokens than the model allows, use continuation with the full previous response as context — not summarization. If you’re hitting stop sequences unexpectedly, log the raw provider request to see what the SDK actually sent.

The symptom is always the same. The cause is never ambiguous if you instrument correctly.