Tokens per second streaming chat is the metric most teams cite when comparing LLM providers, but it is also the most abused. Raw throughput numbers from provider dashboards measure server-side token generation under ideal batch conditions, not the experience a user sees when response text paints onto a screen. If you optimize for the wrong number, you will ship a chat app that looks sluggish even when your benchmark spreadsheet says it is fast.
The gap between server tps and user-perceived speed
A provider advertises “80 tokens per second” on a model card. That figure is usually measured on a warm GPU with a large batch and a fixed output length, ignoring the HTTP layer, proxy buffering, and the browser’s paint cycle. In a real chat app, the user cares about two things: how long until the first character appears, and how smoothly the rest flows.
Time to first token (TTFT) often dominates perceived latency. A model that generates 100 tok/s but takes 2 seconds to emit the first token feels slower than one at 40 tok/s with 200 ms TTFT. Streaming compounds this: if the client receives tokens in bursts because of TCP congestion or a gateway’s buffering, the visual cadence stutters regardless of the underlying tokens per second streaming chat rate.
Network proximity is a silent variable. A model hosted in a single region served to a global user base pays round-trip latency before the first byte. No GPU optimization reduces physical distance.
What you should actually measure
Stop reporting single-number throughput. Capture a distribution, not a point estimate.
TTFT
Measure from request send to the first byte of the SSE chunk containing a token. This includes network round trip, auth, queueing, and prefill. Report median and p95.
Sustained streaming rate
After the first token, count tokens received and divide by elapsed time between first and last token. Ignore the prefill period. This is the real tokens per second streaming chat figure that affects mid-response smoothness.
End-of-stream overhead
The gap between the final token chunk and the done event matters for UI state cleanup. A slow shutdown leaves the typing indicator spinning.
Here is a minimal Python harness using the OpenAI SDK against any OpenAI-compatible endpoint:
import asyncio, time, openai, statistics
async def stream_sample(client, prompt, n=20):
ttfts, rates = [], []
for _ in range(n):
start = time.perf_counter()
first = None
tok_count = 0
async with await client.chat.completions.create(
model="mistral-7b",
messages=[{"role": "user", "content": prompt}],
stream=True,
) as stream:
async for chunk in stream:
if not chunk.choices:
continue
delta = chunk.choices[0].delta.content
if delta:
tok_count += 1
if first is None:
first = time.perf_counter()
ttfts.append(first - start)
last = time.perf_counter()
if first and tok_count > 1:
rates.append((tok_count - 1) / (last - first))
return {
"ttft_median": statistics.median(ttfts),
"ttft_p95": sorted(ttfts)[int(0.95*len(ttfts))-1],
"tps_median": statistics.median(rates),
}Run this from the same network location as your production front end. A number measured from a developer laptop in a different continent is worthless.
Designing a benchmark that resembles production
Synthetic 32-token completions are useless. Real chat prompts are 200–2000 tokens of context, and users send follow-ups. Use a corpus of real conversations from your app or a public dataset. Simulate concurrency: a single streaming request saturates none of the stack; 50 concurrent streams expose proxy limits and provider rate limits.
If you use a gateway that adds fallback, test degradation paths. For example, an OpenAI-compatible endpoint like n4n.ai automatically reroutes when a provider is rate-limited, which keeps TTFT bounded but does not magically increase tokens per second streaming chat on the fallback model. Your benchmark must record which model actually served the response and whether fallback occurred.
{
"request_id": "abc123",
"routed_model": "anthropic/claude-3-haiku",
"fallback_used": true,
"ttft_ms": 412,
"sustained_tps": 58.3,
"total_tokens": 240
}Honor cache-control hints in your client if the gateway forwards them. Repeated system prompts with cache_control: { "type": "ephemeral" } cut prefill time on supporting providers, directly improving TTFT. A realistic benchmark includes a cached-system-prompt run and a cold run.
Concurrency testing is straightforward with asyncio.gather:
async def load_test(client, prompt, concurrency=50):
tasks = [stream_sample(client, prompt, n=2) for _ in range(concurrency)]
return await asyncio.gather(*tasks)Inspect the p95 TTFT across all tasks. If it balloons at 50 concurrent streams, your provider or gateway quota is the bottleneck, not the model’s raw tps.
Tradeoffs: model size, quantization, and routing
Smaller models (7B–13B) typically sustain higher tokens per second streaming chat rates and lower TTFT because they fit in fewer GPUs and require less compute per token. But they hallucinate more on nuanced tasks. A 70B model might run at half the tps yet produce answers that need no follow-up, reducing total interaction time. Evaluate with your own task suite, not MMLU.
Quantization (INT4/INT8) trades a little quality for large tps gains on commodity hardware. Test both: if your eval suite shows <2% regression, ship the quantized variant. The throughput win often offsets the minor quality loss for chat use cases.
Provider routing directives let you pin a request to a specific backend. Use this in benchmarks to isolate variables. Never assume “the endpoint” is one model; a gateway may map a logical name to multiple physical ones. Per-token usage metering, which n4n.ai provides, lets you correlate cost with model choice, but does not alter the throughput you measure.
Client-side rendering is part of the benchmark
Tokens arriving at 60 tps mean nothing if your React component batches state updates every 500 ms. Streaming chat UIs must append deltas incrementally.
function useStreamingText() {
const [text, setText] = useState("");
const buf = useRef("");
useEffect(() => {
const id = setInterval(() => {
if (buf.current) {
setText((t) => t + buf.current);
buf.current = "";
}
}, 16); // ~1 frame
return () => clearInterval(id);
}, []);
return (delta: string) => { buf.current += delta; };
}If you skip the flush interval and only call setText on each SSE message, you inherit the network’s packet boundaries. That produces visible jitter even when the upstream tokens per second streaming chat is smooth.
Also consider backpressure: if the model outpaces your renderer, you should buffer, not drop. But if the buffer grows unbounded, you have a client bug, not a model problem.
Honest limitations of any tps number
Network geography matters. A model hosted in us-east-1 served to a user in Sydney pays 200 ms RTT before the first token. No amount of GPU optimization fixes that. Run benchmarks from the same region as your users.
Batch sizing on the server side means your isolated test might get a dedicated worker while production contends with others. Request providers for shared-tenant numbers or run load tests at expected peak. Single-stream tps is an upper bound, not a prediction.
Takeaway
Benchmark tokens per second streaming chat only as one axis of a multi-dimensional profile: p95 TTFT, median sustained tps, and end-of-stream latency, all measured from the browser under realistic concurrency. Pick the smallest model that meets your quality bar and renders smoothly under load. Use a gateway for resilience and routing control, not as a tps booster. Ship the number that predicts whether a user finishes typing their next message before the answer appears.