A streamed response cuts off early in production far more often than the model actually failing to emit a termination token. The root cause sits in the transport layer: someone between the inference server and your read loop silently dropped the connection, buffered the bytes, or misparsed the Server-Sent Events (SSE) framing. If you are debugging truncated LLM output, stop looking at prompt length and start inspecting every hop that carries text/event-stream.
The SSE contract and what “end token” means
Server-Sent Events are not a proprietary LLM invention. They are an HTTP standard: the server sends text/event-stream, then emits lines prefixed with data: , separated by a blank line (\n\n). OpenAI-compatible chat completions overload this framing—each data: line carries a JSON patch with a token delta, and the stream ends with data: [DONE] or a clean TCP close.
When your streamed response cuts off early, you are missing either the final delta or the [DONE] sentinel. The model likely generated the full sequence; the bytes just didn’t arrive. The four usual suspects are proxies, buffers, clients, and provider failover.
Root cause 1: Idle timeouts in proxies and load balancers
LLM inference is bursty. A model may emit a few tokens, then pause for two seconds while it computes a tool call or runs a long chain-of-thought step. If no bytes cross the wire during that pause, an intermediary with a default idle timeout will hang up.
AWS ALB defaults to 60 seconds, Nginx proxy_read_timeout is 60s, many Kubernetes ingress controllers are 30s. None of these are tuned for LLM streaming out of the box.
Reproduce with a raw client that logs timing:
curl -N -X POST https://api.example.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","stream":true,"messages":[{"role":"user","content":"Write a 2000-word essay"}]}' \
| ts '%H:%M:%S'If the pipe dies at exactly 00:01:00, you have a timeout, not a model problem. Fix it at the edge:
location /v1/ {
proxy_pass http://upstream;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}Set these values deliberately. A 300s timeout covers almost all single-turn generations; for agentic loops you may need websockets instead.
Root cause 2: Middleware buffering breaks chunk boundaries
Even with timeouts fixed, application servers and language frameworks love to buffer. Flask’s dev server streams fine, but put it behind Gunicorn with default settings and the worker may accumulate 4KB or 8KB before flush. Express with res.write() can be intercepted by compression middleware that waits for the stream to end.
A minimal Python server that respects SSE:
from flask import Response, stream_with_context
import time, json
@app.route('/stream')
def stream():
def gen():
for i in range(5):
yield f"data: {json.dumps({'chunk': i})}\n\n"
time.sleep(2)
return Response(stream_with_context(gen()), mimetype='text/event-stream')If you front this with Nginx, you must also disable proxy buffering as shown above. Without proxy_buffering off;, Nginx will hold the response and deliver it as one blob—or drop it when the client gives up.
The same applies to Node:
app.get('/stream', (req, res) => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
let i = 0;
const id = setInterval(() => {
res.write(`data: ${JSON.stringify({ chunk: i++ })}\n\n`);
if (i > 5) { clearInterval(id); res.end(); }
}, 2000);
});If compression() is mounted globally, it will buffer. Mount it per-route or exclude /stream.
Root cause 3: Client read loops that bail on empty lines or JSON errors
The most common application bug is a client that assumes each read() returns exactly one SSE event. TCP does not work that way. A single data: ...\n\n can be split across two packets, or two events can arrive in one packet. If your parser splits on \n and discards incomplete lines, you silently drop data and may exit the loop when buffer is empty after a partial read.
Here is a correct TypeScript reader that accumulates across chunks:
const res = await fetch('/v1/chat/completions', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ model: 'gpt-4o', stream: true, messages: [{ role: 'user', content: 'hi' }] })
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
let sep: number;
while ((sep = buffer.indexOf('\n\n')) !== -1) {
const rawEvent = buffer.slice(0, sep);
buffer = buffer.slice(sep + 2);
const line = rawEvent.trim();
if (!line.startsWith('data:')) continue;
const data = line.slice(5).trim();
if (data === '[DONE]') return;
try {
const json = JSON.parse(data);
processDelta(json);
} catch (e) {
// Partial JSON from a fragmented chunk: keep rawEvent in buffer instead
buffer = rawEvent + '\n\n' + buffer;
break;
}
}
}Notice the catch block re-injects the raw event into the buffer. That single recovery path eliminates a class of bugs where a streamed response cuts off early because one JSON.parse threw and the surrounding code treated it as fatal.
Root cause 4: Provider degradation and fallback masks
When you call an inference gateway that aggregates multiple upstream model providers, a rate limit or 503 from one provider can trigger a transparent fallback. n4n.ai exposes an OpenAI-compatible endpoint across 240+ models and will automatically fall back when a provider is rate-limited or degraded. If the fallback occurs mid-stream, the gateway may open a new upstream connection and continue emitting tokens—but only if your client tolerates a brief connection reset and reconnects with the correct Last-Event-ID.
Most naive clients treat a sudden done: true from reader.read() as end-of-stream. They never see the continuation because they didn’t implement the SSE retry spec. The symptom looks identical to truncation: your streamed response cuts off early, the UI shows half a sentence, and the server logs show a successful 200.
To defend against this, emit a client-side heartbeat: if no byte arrives in 15s and you haven’t seen [DONE], reopen the request with the last processed event id. Gateways that honor routing directives will resume generation from the fallback provider.
Root cause 5: Partial JSON and token boundaries
Some SDKs parse each data: line immediately and assume it is complete. While OpenAI-compatible streams are well-behaved, self-hosted vLLM or llama.cpp instances sometimes emit whitespace or split a UTF-8 multibyte character across chunks. If your decoder uses decode(value) without { stream: true }, surrogate pairs break and JSON.parse throws.
Always use streaming decode. Always validate that the line ends with } or ] before parsing. If not, wait for more bytes.
import json
def parse_sse_lines(raw: str):
for line in raw.splitlines():
if line.startswith("data: "):
payload = line[6:].strip()
if payload == "[DONE]":
return "done"
try:
return json.loads(payload)
except json.JSONDecodeError:
# hold for next chunk
return "partial"Tradeoffs: reliability vs latency
Disabling buffering and extending timeouts makes streams robust but increases the number of small packets and keeps connections open longer. For a high-traffic consumer app, that can mean more load balancer connections and higher memory per request.
Client-side reconnection adds code complexity and can double-charge tokens if the gateway does not dedupe on resume. You trade a little latency for a lot of resilience.
If you are building internal tooling, lean toward aggressive timeouts and simple clients. If you are serving end users, invest in the buffered reader and reconnect logic—the cost of a truncated response in a chat UI is a lost user.
Decisive takeaway
A streamed response cuts off early is an infrastructure bug, not a model limitation. Audit your stack in this order: (1) confirm the raw curl -N stream completes; (2) check proxy idle timeouts and buffering; (3) verify your server framework flushes per event; (4) rewrite the client to accumulate buffers and reconnect on idle. Do those four and you will eliminate virtually every truncation report.
If you use a gateway that performs provider fallback, treat mid-stream resets as expected behavior and code for resume. The engineers who ship reliable LLM features are the ones who respect the SSE contract instead of assuming the network is a function call.