Shipping a responsive AI chat UI demands hard numbers on streaming latency for chat applications, not vibes. This guide lays out a concrete path to benchmark token delivery across models and providers so you can catch regressions before users feel them.
Define the Metrics That Matter
Streaming latency for chat applications breaks into three distinct measurements:
- Time to first token (TTFT): from request send to first content delta.
- Inter-token latency (ITL): gap between consecutive tokens.
- Total completion time: from send to final delta including finish reason.
Averages lie. A 200 ms TTFT with a 5-second stall mid-stream ruins UX even if the mean looks fine. Track p50, p95, and p99 for each metric.
Build a Minimal Streaming Harness
Use the OpenAI Python SDK against any OpenAI-compatible endpoint. Keep the harness dumb: no retries, no UI, just timestamps.
import asyncio
import time
from openai import AsyncOpenAI
client = AsyncOpenAI(
base_url="https://your-endpoint/v1",
api_key="sk-test",
)
async def measure(prompt: str, model: str):
start = time.perf_counter()
ttft = None
prev = None
gaps = []
async with client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True,
) as stream:
async for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
now = time.perf_counter()
if ttft is None:
ttft = now - start
prev = now
else:
gaps.append(now - prev)
prev = now
return {"ttft": ttft, "p95_gap": sorted(gaps)[int(len(gaps)*0.95)] if gaps else None}
async def main():
print(await measure("Explain QUIC in one paragraph.", "gpt-4o-mini"))
asyncio.run(main())Run this from the same network location as your production app. Client-side measurement captures DNS, TLS, and proxy overhead that backend logs miss.
Control Model, Provider, and Routing
Latency varies wildly by model size and provider health. Fix the model and region for each test run. If you route through a gateway such as n4n.ai, note that it honors client routing directives and forwards provider cache-control hints; include those headers in your test to mimic production. Its automatic fallback when a provider is degraded will mask outages but also change latency profiles, so disable fallback for pure benchmarking.
Send a routing header if your gateway supports it:
{
"x-n4n-route": "provider:azure;region:eastus",
"cache-control": "max-age=300"
}Without explicit routing you are measuring a moving target.
Capture Server-Side Signals When Possible
Client timestamps show user-perceived latency. Server logs show where time goes. If the inference endpoint emits per-token timestamps in response headers or SSE comments, record them.
Common SSE stream from a compliant server:
data: {"choices":[{"delta":{"content":"Hello"}}]}
data: {"choices":[{"delta":{"content":" world"}}]}Some gateways inject a comment with queue time:
: queue_ms=12 gen_ms=3Parse these to separate scheduling delay from generation delay.
Simulate Real Concurrency
A single stream hides saturation behavior. Run N concurrent streams with a realistic prompt mix. Use asyncio.gather with a semaphore to cap parallelism.
async def load_test(model: str, prompts: list[str], concurrency: int):
sem = asyncio.Semaphore(concurrency)
async def bounded(p):
async with sem:
return await measure(p, model)
return await asyncio.gather(*(bounded(p) for p in prompts))Start at 1, then 5, 10, 25, 50. Plot TTFT p95 against concurrency. Most providers degrade gracefully until a knee, then TTFT explodes.
Account for Prompt and Output Length
Token count drives generation time. Benchmark with three buckets:
- Short prompt, short answer (<50 tokens)
- Long system prompt, medium answer (500 tokens)
- Long prompt, long answer (2000+ tokens)
Streaming latency for chat applications is not constant per token. Prefill cost scales with input tokens; decode cost scales with output. Separate the two by measuring TTFT (prefill) versus tokens-per-second after first byte.
Avoid Measurement Artifacts
Several traps skew results:
TCP/TLS Reuse
Opening a new connection per request adds 50–200 ms. Use a persistent AsyncOpenAI client across calls; it pools connections.
Client Buffering
Some proxies buffer SSE until a chunk size threshold. Set stream=True and disable any middleware that buffers. Test with curl --no-buffer to confirm deltas arrive live.
Clock Skew
If comparing client and server logs, sync with NTP. A 100 ms offset makes ITL analysis worthless.
Tokenizer Mismatch
Counting Python string splits is not token counting. Use the provider’s tokenizer or tiktoken for OpenAI models to compute true tokens/sec.
Analyze Percentiles and Jitter
Dump raw measurements to CSV. Compute:
import statistics
def pct(data, p):
s = sorted(data)
return s[min(len(s)-1, int(len(s)*p))]Plot TTFT p50/p95/p99 and ITL p99. Jitter—the variance in ITL—matters more for chat feel than raw speed. A steady 30 ms/token beats a 10 ms average with 200 ms stalls.
Track Over Time in CI
Latency creeps. Wire the harness into a nightly job. Store results in a time-series store and alert when p95 TTFT rises >20% week-over-week.
# github actions snippet
- name: Benchmark latency
run: python bench_stream.py --model gpt-4o-mini --out metrics.json
- name: Upload
uses: actions/upload-artifact@v4
with:
name: latency-metrics
path: metrics.jsonIf you ship a chat app, treat streaming latency for chat applications as a documented SLO, not a mystery.
Tradeoffs When Optimizing
Reducing TTFT often means smaller models or speculative decoding, which can hurt answer quality. Lower ITL may require higher batch occupancy on the server, increasing cost per token. Measure both quality (via eval) and latency together; never tune one blind to the other.
If you must pick, prioritize TTFT and ITL p99 over average throughput. Users perceive the worst stall, not the mean.