Streaming latency in an LLM API is the delay a client observes from sending a request to receiving the first generated token, plus the irregular gaps between subsequent tokens. A streaming latency LLM API therefore defines user-perceived responsiveness not by total completion time alone, but by how quickly and smoothly tokens arrive.
What streaming latency actually measures
Two numbers dominate the experience:
- Time to first token (TTFT): wall-clock from request send to the first delta chunk containing content.
- Inter-token latency (ITL): median or p95 gap between consecutive token chunks during decode.
A third metric, tail TTFT, matters for degraded conditions. When evaluating a streaming latency LLM API, isolate TTFT from ITL—they come from different phases of the inference pipeline.
Total latency is TTFT + (N-1) * ITL for N tokens. That formula is trivial, but most teams optimize the sum while users feel the parts separately.
How token streaming works under the hood
A client sends a standard chat completion request with "stream": true. The server runs prefill (processing the prompt to build the KV cache), then enters decode (autoregressive token generation). Each token is serialized as a chunk and pushed over the open connection.
Transport: SSE over HTTP
OpenAI-compatible APIs use Server-Sent Events over chunked HTTP. A raw line looks like:
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"The"}}]}The client reads lines prefixed with data: , parses JSON, and extracts choices[0].delta.content. The stream ends with data: [DONE].
Server-side scheduling
Modern inference servers use continuous batching. Prefill for your request may wait behind others; that wait is pure TTFT penalty. Decode is usually interleaved across many requests, so your ITL depends on batch size and scheduler fairness, not just raw model speed.
Provider buffering is a silent killer. Some implementations accumulate a few tokens before flushing to reduce syscall overhead. That trades ITL variance for worse TTFT or bursty delivery.
Why streaming latency matters more than total latency
In a non-streaming call, the user stares at a spinner until the whole answer exists. Streaming converts wait time into reading time. But if TTFT exceeds ~300 ms, the interface feels broken before the first character appears. If ITL spikes beyond ~100 ms mid-sentence, the text appears to stutter.
Most benchmarks for a streaming latency LLM API report median TTFT because that is the gate to perceived interactivity. A 2-second total latency with 200 ms TTFT and smooth ITL feels faster than a 1.5-second total with 1.2 s TTFT.
A concrete measurement example
Below is a minimal Python client that measures TTFT and per-token ITL against any OpenAI-compatible endpoint. It uses requests with stream=True and a monotonic clock.
import requests, time, json
url = "https://api.openai.com/v1/chat/completions"
headers = {"Authorization": "Bearer $KEY", "Content-Type": "application/json"}
payload = {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Explain latency"}],
"stream": True,
}
start = time.perf_counter()
first_token_ts = None
prev_ts = None
token_count = 0
with requests.post(url, headers=headers, json=payload, stream=True) as r:
for line in r.iter_lines():
if not line or not line.startswith(b"data: "):
continue
data = line[6:].strip()
if data == b"[DONE]":
break
chunk = json.loads(data)
delta = chunk["choices"][0].get("delta", {}).get("content")
if delta:
now = time.perf_counter()
if first_token_ts is None:
first_token_ts = now
print(f"TTFT: {(first_token_ts - start) * 1000:.1f} ms")
else:
print(f"ITL: {(now - prev_ts) * 1000:.1f} ms")
prev_ts = now
token_count += 1Run this against your provider and plot the ITL histogram. You will usually see a tight cluster with occasional multi-hundred-ms outliers caused by scheduler preemption or network retransmission.
The final chunk often carries usage:
{
"choices": [{"delta": {}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 12, "completion_tokens": 34, "total_tokens": 46}
}Per-token usage metering requires summing deltas or reading this final frame; do not wait for a non-existent separate invoice call.
Common misconceptions
“Low total latency implies low streaming latency”
A provider can prefill slowly but decode fast. You get a long blank pause, then a rapid dump. Total looks fine; UX is poor. Always split the metrics.
“All providers stream identically”
They do not. Some send each token as a separate TCP segment; some coalesce under Nagle’s algorithm. Some emit whitespace as its own delta; some fold it into the next word. A proxy that buffers 4 KB before forwarding will wreck TTFT even if the origin is fast.
“Token count is the only variable”
Network path, TLS record sizing, and HTTP/2 vs HTTP/1.1 framing change observed latency. A gateway that terminates TLS and re-opens a backend connection adds at least one round trip. Client-side iter_lines() parsing in Python adds milliseconds if you allocate excessively.
“Streaming removes the need for fallback”
If a provider is rate-limited mid-stream, the connection may hang. An OpenAI-compatible gateway like n4n.ai, which exposes one endpoint across 240+ models with automatic fallback, must proxy chunks immediately and honor client routing directives so a degraded upstream does not silently stall the stream. That design choice is invisible but determines whether your TTFT survives a provider outage.
Practical implications for API design
If you operate a service in front of model providers:
- Set
X-Accel-Buffering: noon upstream responses if behind nginx. - Never
json.dumpsan entire accumulated string before sending; forward deltas. - Forward
cache-controlhints from the origin. Provider cache hits reduce TTFT dramatically; stripping those hints forces recompute. - Measure TTFT and p95 ITL from the edge, not just from the backend, because proxy hops are part of the streaming latency LLM API contract your users experience.
For clients:
- Open the stream with a timeout on the first byte, not just on the whole response.
- Render tokens incrementally; do not wait for sentence boundaries.
- Treat ITL spikes as a signal to show a subtle “still working” indicator rather than freezing the UI.
Streaming latency is an end-to-end property. The model is only one component. The moment you treat TTFT and ITL as distinct, measurable, and architecturally sensitive quantities, your interface stops feeling slow.