You wired up an LLM call with stream: true, but the client receives the entire response in one lump after a long delay. This streaming response buffering not flushing behavior is almost never a bug in the model endpoint; it is an artifact of the layers sitting between your code and the raw token stream.
The illusion of streaming
Streaming an HTTP response means sending bytes to the client as they become available. In an LLM context, those bytes are Server-Sent Events (SSE) frames carrying tokens. When you observe a frozen spinner for ten seconds followed by a wall of text, the tokens were generated incrementally upstream but got parked somewhere downstream.
The root cause is that every intermediate component defaults to batching. Batching reduces syscall overhead, improves compression ratio, and simplifies error handling. The cost is exactly the latency you were trying to avoid by streaming. Understanding where the bytes stall is the difference between a snappy chat UI and a confused user.
Where bytes actually stall
Application framework buffering
Most web frameworks buffer output unless you explicitly opt into streaming. Django, Rails, and even Node’s default res.write can sit on data if the response object isn’t flushed.
In Python, using requests without stream=True will read the whole body:
import requests
r = requests.get("https://api.example.com/v1/chat/completions", stream=False)
# r.text blocks until the connection closesThe fix is trivial but easy to miss:
r = requests.get(url, stream=True)
for line in r.iter_lines():
if line:
print(line)Flask requires Response(stream_with_context(...)) or stream=True on the route. FastAPI with StreamingResponse works, but if you wrap it in a middleware that reads the body, you lose chunks. A common mistake is attaching a logging middleware that calls await response.body() — that consumes the stream and forces a buffer.
If you emit tokens from a Python process via print, stdout is line-buffered when connected to a terminal but fully buffered when piped to a socket or file. You must flush:
print(token, end="", flush=True) # mandatory for live outputProxy and gateway buffering
Reverse proxies are the usual suspects. Nginx ships with proxy_buffering on by default. It will accumulate up to proxy_buffer_size before sending to the client.
location /api/ {
proxy_pass https://upstream;
proxy_buffering off; # disable to flush immediately
proxy_cache off;
}If you cannot change proxy config, send X-Accel-Buffering: no from your app.
Gateways that aggregate multiple providers introduce another wrinkle. Even a lean gateway such as n4n.ai, which forwards provider streams and honors client routing directives, may briefly buffer during an automatic fallback event when a primary provider is rate-limited or degraded. The buffer protects the client from a broken stream, but it adds a visible stall.
CDNs and load balancers often buffer SSE as well. Some default to a 1–2 second coalescing window to optimize edge delivery. If curl -N directly to your origin streams fine but the same request through the CDN lumps, you have your answer.
Compression and content-encoding
If the response is gzipped, the compressor needs a window. gzip with default settings will hold data until the window fills or the stream ends. For SSE, you must either disable compression or use chunked transfer with Content-Encoding: identity and compress per-chunk (rare).
# force uncompressed stream from curl
curl -N -H "Accept-Encoding: identity" https://api.example.com/streamBrotli and zstd are worse for latency because they look further ahead. If you see buffering only when Accept-Encoding: br is present, that’s your culprit. The fix is to carve out a no-compression rule for streaming routes:
map $request_uri $no_compress {
~^/v1/stream 1;
default 0;
}
gzip off if ($no_compress);TCP and kernel socket buffers
Even with every app-layer flush correct, the Linux kernel may coalesce small writes. TCP_NODELAY disables Nagle’s algorithm, which waits for more data to fill a segment.
import socket
sock = socket.socket()
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)In practice, kernel buffering is measured in milliseconds, not seconds. If your stall is multi-second, look higher. Still, for high-frequency token streams (multiple per millisecond), enabling TCP_NODELAY on the gateway socket removes a measurable tail latency.
Client-side rendering throttles
A server that flushes perfectly can still appear buffered in the browser. React state updates inside an onmessage handler that fires 50 times per second will batch renders unless you use a queuing approach or requestAnimationFrame. The stream is live; the paint is not.
let buffer = "";
source.onmessage = (e) => {
buffer += e.data;
// defer paint to next frame
requestAnimationFrame(() => setTokens(buffer));
};Concrete debugging steps
Start at the edge and move inward. Use curl -N to bypass your app entirely:
curl -N -i https://your-domain.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"stream":true,"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'If curl -N streams token-by-token, your server or client library is buffering. If it still lumps, the proxy or gateway is guilty.
Next, reproduce from a bare Node script:
const res = await fetch("https://your-domain.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ stream: true, model: "gpt-4o", messages: [{ role: "user", content: "hi" }] }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
process.stdout.write(decoder.decode(value));
}If Node streams but your React app doesn’t, the browser’s fetch is fine; your state update logic is throttling renders.
Add a timestamped probe endpoint. Emit a byte every 100 ms with zero middleware:
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import time, asyncio
app = FastAPI()
@app.get("/debug/stream")
async def debug_stream():
async def gen():
while True:
yield f"data: {time.time()}\n\n"
await asyncio.sleep(0.1)
return StreamingResponse(gen(), media_type="text/event-stream")Walk this endpoint through each layer: local process, origin server, proxy, CDN. The first hop that delays the ticks is your offender.
Tradeoffs of forcing flush
Disabling buffering everywhere is not free.
- Throughput: Smaller writes mean more syscalls. At 1000 concurrent streams, CPU rises noticeably.
- Compression: Turning off gzip on a 10KB token stream wastes bandwidth but saves 20–50 ms latency per chunk.
- Proxy memory: With
proxy_buffering off, Nginx uses more temporary space per connection; tuneproxy_temp_file_write_sizeto avoid disk spills. - Error handling: If you flush immediately and the upstream dies mid-stream, the client gets a truncated response with no HTTP error code. SSE requires you to send a custom
[DONE]event; buffering lets the proxy return 502 cleanly. - Observability: Per-token logging becomes noisy. You may need sampling.
A pragmatic setup: keep compression off for /v1/stream routes, set proxy_buffering off only there, and use TCP_NODELAY on the gateway socket. That isolates the cost to the one path where latency matters.
Decisive takeaway
When you see streaming response buffering not flushing, blame the plumbing, not the model. Audit proxies first (Nginx, CDN, gateway fallback), then framework middleware, then compression. Use curl -N as your ground truth. Only after those are clean should you profile kernel sockets or client render loops.
The decisive move: create a /debug/stream endpoint that emits a timestamped byte every 100 ms with no middleware, and watch it through each layer. The first hop that delays the ticks is your offender. Fix that, and your tokens will flow like they should. The streaming response buffering not flushing problem is solved by removing batching at exactly one layer—find it with measurement, not guesswork.