How streaming affects time-to-first-token benchmarks

Most latency reports treat time-to-first-token as a single static number, but enabling streaming changes both what you measure and how the number behaves under load. If you care about responsive UX, you need to benchmark streaming time to first token the way your application actually calls the model, not via a one-shot completion. The gap between a non-streaming request and a streamed one can hide seconds of perceived latency.

What streaming changes in the request path

Streaming shifts the server from “generate everything, then send” to “send each token as it finishes.” That sounds like it should make time-to-first-token identical to the non-streaming case, because the model still needs to process the prompt before emitting the first token. In practice, the streaming path adds SSE framing, flush calls, and often a different scheduling class on the inference server.

When you call a non-streaming endpoint, the gateway or provider typically buffers the full response, computes usage, and returns one JSON blob. With streaming, the provider opens a chunked HTTP response and writes data: {...}\n\n lines as tokens are ready. The first chunk containing a content delta is your true streaming time to first token. Everything before that—TLS handshake, request routing, prompt prefill—is identical, but the measurement surface is noisier.

Measure streaming time to first token on the client

Timestamp before send and on first delta

Do not rely on server-reported timestamps unless you control the server clock and the network path. Client-side perf_counter is the only honest source for UX latency.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain TCP slow start in one paragraph"}],
    stream=True,
    temperature=0,
)
ttft = None
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        ttft = time.perf_counter() - start
        print(f"streaming time to first token: {ttft*1000:.1f} ms")
        break
# close the stream to free connection
stream.close()

The break stops iteration after the first token, but some SDKs require you to exhaust or close the stream. In the OpenAI SDK, stream.close() works.

Ignore SSE comments and ping frames

Gateways often send : keep-alive or : ping lines to prevent proxy timeouts. These are not tokens. Parse only lines starting with data: and containing a non-empty choices[0].delta.content.

Build a reproducible benchmark harness

Control every variable you can

A clean streaming time to first token benchmark fixes:

• Model ID (exact snapshot)
• Prompt length and content (use a static string)
• Sampling params (temperature=0, top_p=1)
• Client geography (same AZ or city)
• SDK version

Run at least 20 measurements after 3 warm-up calls. Report median and p95, never a bare mean.

Raw HTTP probe to bypass SDK overhead

SDKs add serialization cost. For a lower-bound number, use curl with -N (no buffering) and capture time to first byte, then refine with a small parser.

curl -N -s -X POST https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"stream":true}' \
  -o /dev/null -w "ttfb=%{time_starttransfer}\n"

time_starttransfer is the closest curl gets to TTFT, but it includes response headers. To get exact token latency, pipe through a line parser:

curl -N -s ... | grep -m1 -o '"content":"[^"]*"' | head -c 50

Account for network and buffering effects

Proxy buffering hides real latency

Nginx and many CDNs buffer chunked responses by default. If your benchmark runs behind a misconfigured proxy, the first byte you receive is the whole buffered body, not the first token. Send X-Accel-Buffering: no if you control the edge, or benchmark against the origin.

Connection reuse and cold TLS

A fresh connection adds a TLS handshake (50–150 ms depending on cipher and RTT). Use a persistent requests.Session or connection pool. Measure both cold and warm connection TTFT separately.

import requests, time, json

session = requests.Session()
url = "https://api.example.com/v1/chat/completions"
headers = {"Authorization": "Bearer "+KEY, "Content-Type": "application/json"}
data = {"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"stream":True}

start = time.perf_counter()
r = session.post(url, headers=headers, json=data, stream=True)
for line in r.iter_lines():
    if line and b"content" in line and b"delta" in line:
        ttft = time.perf_counter() - start
        print(f"ttft={ttft*1000:.1f}ms")
        break
r.close()

Factor in provider fallback and routing

If your stack calls a gateway that abstracts multiple providers, automatic fallback can distort numbers. Suppose the primary provider is rate-limited; the gateway retries a secondary, adding a full round-trip before any token streams. That is not model latency, it is routing overhead.

A gateway such as n4n.ai provides one OpenAI-compatible endpoint for 240+ models and applies automatic fallback when a provider is degraded. When benchmarking streaming time to first token, pin a specific provider via routing headers or disable fallback, and log when a retry occurs. Honoring client routing directives lets you isolate the model you intend to measure.

Common pitfalls

Mistaking time-to-first-byte for token latency

We already noted this. Concretely, a 200 OK with empty SSE comments yields a TTFB of ~20 ms while the real first token arrives at 800 ms. Always parse the payload.

Ignoring prompt caching

Providers with prefix caching (e.g., Anthropic, OpenAI) return dramatically lower TTFT on repeated prefixes. Label cold vs warm. If your gateway forwards provider cache-control hints, warm runs can be 5–10x faster. Mixing them ruins the benchmark.

Averaging across model classes

A 7B model on a cheap GPU might hit 50 ms TTFT; a 70B model on shared infra might be 600 ms. Averaging them produces a number that describes neither. Slice by model.

Client-side processing delay

In Python, a slow for loop or logging per chunk can delay detection of the first token by tens of ms. Use async or minimize work inside the loop.

import asyncio, time
from openai import AsyncOpenAI

async def measure():
    client = AsyncOpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
    start = time.perf_counter()
    stream = await client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role":"user","content":"hi"}], stream=True)
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - start

Tradeoffs: streaming vs non-streaming benchmarks

Streaming benchmarks reflect what users feel in a chat UI: they see characters appear quickly. Non-streaming benchmarks are easier to automate and give a clean total generation time. For capacity planning you need both: TTFT for interactivity, total latency and tokens/sec for throughput.

If you only report streaming time to first token, you may miss cases where the provider streams the first token fast but then stalls. Always capture the full stream duration alongside TTFT.

Actionable ordered path

1. Define the metric – first content delta, not first byte, not completion.
2. Pin the environment – model, prompt, params, region, SDK.
3. Warm up – 3 throwaway calls to trigger caches and JIT paths.
4. Run ≥20 iterations – record client-side perf_counter deltas.
5. Disable buffering – set X-Accel-Buffering: no or hit origin.
6. Separate cold/warm cache – run two distinct suites.
7. Pin routing – if using a gateway, disable fallback or tag provider.
8. Report median and p95 – with explicit model and date.

Follow that and your streaming time to first token numbers will survive scrutiny from other engineers who actually ship latency-sensitive LLM features.