Measuring time to first token in production API calls

Measuring time to first token (TTFT) in production API calls separates responsive LLM features from ones that feel broken. If you treat measuring time to first token as a one-off curl test, you will miss queue delays, cold starts, and provider degradation that only show under real traffic.

Step 1: Define the measurement boundary

TTFT is the elapsed time from the instant the client sends the HTTP request to the instant the first byte of the response body reaches the client socket. Use a monotonic clock (time.monotonic() in Python, process.hrtime() in Node) so wall-clock adjustments don’t corrupt deltas.

Exclude application-level pre-processing (building the prompt) from the measurement. Include TLS handshake and connection setup unless you are specifically benchmarking warm connections. When measuring time to first token across a fleet, lock the definition once and enforce it in code.

What counts as “first token”

For streaming chat completions, the first token is the first chunk where choices[0].delta.content is non-empty. SSE comments (": keep-alive") and empty data: lines are not tokens.

Step 2: Instrument a streaming client

The OpenAI Python SDK supports streaming. Capture the start time immediately before create, then break on the first content delta. Close the stream to avoid leaking connections.

import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

async def measure_ttft(model: str, prompt: str) -> float:
    start = time.monotonic()
    first_ts = None
    stream = await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            first_ts = time.monotonic()
            break
    await stream.close()
    return (first_ts - start) if first_ts else -1.0

If you run this in a loop, reuse the AsyncOpenAI client. Creating a new client per call adds TLS overhead that pollutes TTFT.

Step 3: Eliminate client-side buffering artifacts

Higher-level SDKs may buffer chunks or decode SSE into objects before yielding. To be certain you are seeing raw network latency, drop to httpx and iterate raw bytes.

import httpx, time, json

def measure_ttft_raw(url: str, headers: dict, payload: dict) -> float:
    start = time.monotonic()
    first_ts = None
    with httpx.Client(timeout=30) as c:
        with c.stream("POST", url, headers=headers, json=payload) as r:
            for raw in r.iter_raw():
                if raw:
                    first_ts = time.monotonic()
                    break
    return first_ts - start if first_ts else -1.0

headers = {
    "Authorization": "Bearer sk-...",
    "Content-Type": "application/json",
}
payload = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "hi"}],
    "stream": True,
}
print(measure_ttft_raw("https://api.openai.com/v1/chat/completions", headers, payload))

iter_raw bypasses line splitting and SSE parsing. If your TTFT from Step 2 and Step 3 disagree by more than a few milliseconds, suspect SDK buffering.

Step 4: Sample across models and providers

A single provider’s TTFT varies by model, region, and load. Using a single OpenAI-compatible endpoint such as n4n.ai’s, which fronts 240+ models with automatic fallback, lets you run identical measurement code while varying only the model string. That removes client-side variance from key management and base-URL switching.

models = ["gpt-4o-mini", "claude-3-haiku", "mistral-large"]
for m in models:
    ttft = asyncio.run(measure_ttft(m, "Explain TTFT briefly."))
    print(m, f"{ttft*1000:.1f}ms")

Run each model at least 50 times spread across minutes. Provider queues breathe; a single sample is anecdote, not data.

Step 5: Capture server-reported timings

Some gateways expose processing hints in response headers. Read them before consuming the body. They are useful as a cross-check, not a replacement for socket measurement.

with httpx.Client(timeout=30) as c:
    with c.stream("POST", url, headers=headers, json=payload) as r:
        server_hint = r.headers.get("x-ttft-ms")  # example header
        for raw in r.iter_raw():
            if raw:
                client_ttft = (time.monotonic() - start) * 1000
                break

If the gateway forwards provider cache-control hints or routing directives, set x-routing-directive: no-fallback during measurement so automatic failover doesn’t inflate TTFT on a degraded primary.

Step 6: Persist and aggregate

Write every sample to JSONL. Keep the model, timestamp, TTFT in milliseconds, and any routing tag.

import json, time as _t

def log_sample(model: str, ttft_ms: float, tag: str = ""):
    with open("ttft.jsonl", "a") as f:
        f.write(json.dumps({
            "model": model,
            "ttft_ms": round(ttft_ms, 2),
            "epoch": _t.time(),
            "tag": tag,
        }) + "\n")

Aggregate with stdlib to avoid dependency bloat:

import json, statistics

rows = [json.loads(l) for l in open("ttft.jsonl")]
by_model = {}
for r in rows:
    by_model.setdefault(r["model"], []).append(r["ttft_ms"])

for model, vals in by_model.items():
    vals.sort()
    p50 = statistics.median(vals)
    p95 = vals[int(len(vals)*0.95)-1]
    print(f"{model}: p50={p50:.0f}ms p95={p95:.0f}ms n={len(vals)}")

When measuring time to first token at scale, plot p50/p95 over time. A rising p95 with stable p50 means tail latency from queueing, not a regression in your code.

Step 7: Validate your pipeline

Before trusting numbers, point your harness at a mock server with a known delay.

from flask import Flask, Response
import time

app = Flask(__name__)

@app.route("/v1/chat/completions", methods=["POST"])
def mock():
    def gen():
        time.sleep(0.2)  # deterministic 200ms delay
        yield b'data: {"choices":[{"delta":{"content":"hi"}}]}\n\n'
    return Response(gen(), mimetype="text/event-stream")

if __name__ == "__main__":
    app.run(port=5000)

Run measure_ttft_raw against http://localhost:5000/v1/chat/completions. A correct implementation reports ~200ms ± a few ms. If you see 0ms or 400ms, your start/stop points are wrong.

Step 8: Avoid common pitfalls

Cold connections. The first call in a process pays TLS and DNS. Warm the client with one discarded request per process before collecting samples.

Proxy buffering. nginx and some load balancers buffer SSE by default. Set proxy_buffering off; or measure from inside the same network segment as the gateway.

SDK timeouts masquerading as TTFT. If the stream opens but the first token never arrives, your code should record a timeout, not a negative or zero value.

Fallback hidden in the path. Automatic provider fallback is great for uptime but poisonous for benchmarks. Tag measurement traffic to disable it, or subtract the known redirect cost.

Mixing definitions. Don’t compare TTFT measured at the socket with TTFT measured at the UI layer. They differ by network egress and parse time. Pick one boundary and stamp it in your logging.

Verify success

You have a working TTFT benchmark when:

1. The mock server test yields the injected delay within 5ms.
2. Production samples show p50/p95 stable across repeated runs on the same model.
3. Raw-byte measurement and SDK measurement agree within 10ms.
4. Every record includes the exact model string and routing tag.

After that, measuring time to first token becomes a routine signal in your latency dashboard, not a fire drill when users complain about lag.