Time-to-first-token benchmark for reasoning models

Time to first token reasoning models has become the latency metric teams cite when comparing o1-class systems, yet most measurements conflate network handshake with hidden inference. A reasoning model may spend tens of seconds generating internal tokens before emitting anything visible, and that cost is exactly what time to first token should capture. If you benchmark it like a standard chat model, you will mis-size your timeout budgets and ship a broken loading state.

What TTFT actually measures for reasoning models

Standard LLM APIs return the first token a few hundred milliseconds after request receipt. Reasoning models invert that expectation: they run a constrained search or multi-step self-dialogue server-side, then surface a synthesized answer. The first streamed byte might be a role marker, a <thinking> delimiter, or the opening word of the final response.

Define TTFT as the interval from sending the final request byte to receiving the first chunk containing delta.content or equivalent. For OpenAI-compatible streams, that is the first event where choices[0].delta.content is non-empty.

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.example.com/v1")
start = time.perf_counter()
stream = client.chat.completions.create(
    model="reasoning-model",
    messages=[{"role": "user", "content": "Prove sqrt(2) irrational"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft = time.perf_counter() - start
        print(f"TTFT: {ttft:.2f}s")
        break

This looks simple, but the value you get depends entirely on whether the provider streams intermediate reasoning.

Why naive benchmarks mislead

Run the same loop against three providers and you will see TTFT ranging from under a second to tens of seconds for the identical prompt. The spread is not hardware alone; it is streaming policy.

Some providers buffer the entire reasoning trace and emit it as one chunk after completion. Their “TTFT” equals total generation latency. Others stream the reasoning tokens as they are produced, so TTFT is near the start of the hidden phase. Comparing those numbers directly is nonsense.

A curl measurement exacerbates this:

curl -w "time_starttransfer: %{time_starttransfer}\n" \
  -d '{"model":"reasoning-model","messages":[{"role":"user","content":"hi"}],"stream":true}' \
  -H "Authorization: Bearer $KEY" https://api.example.com/v1/chat/completions

time_starttransfer captures the first byte, which could be a Server-Sent Events comment (\n\n) sent to keep the connection alive. You have measured keep-alive, not token generation.

Parse the actual token

Always decode the SSE frame and check for content. In Python, the openai SDK does this, but if you use requests or aiohttp directly, skip empty data: lines and heartbeat comments.

Methodology for honest measurement

To get defensible numbers for time to first token reasoning models, control three variables: prompt complexity, cache state, and streaming mode.

Isolate network from compute

Warm up the connection with a trivial request. Then send the real prompt and record perf_counter at the last byte written and at first content delta. Run 20 iterations, discard the first five. Report median and p95, not average.

Account for provider buffering and caching

Prefix caching changes TTFT dramatically. A cached system prompt skips reprocessing thousands of tokens. If your gateway forwards cache-control hints, you can test both states without code changes. For example, a gateway such as n4n.ai that honors client routing directives and forwards provider cache-control hints lets you send the same request body with cache_control set and observe the delta.

{
  "model": "reasoning-model",
  "messages": [
    {"role": "system", "content": "You are a math tutor", "cache_control": {"type": "ephemeral"}},
    {"role": "user", "content": "Solve the Basel problem"}
  ],
  "stream": true
}

Without caching, TTFT includes prompt prefill. With caching, it drops to the reasoning startup cost. Both are valid, but you must label them.

Sample diverse prompts

Reasoning time scales with problem difficulty. Benchmarking only “What is 2+2?” yields optimistic TTFT. Include a multi-step coding task and a logic puzzle. The variance matters more than the mean.

Tradeoffs: streaming vs polling

Streaming is mandatory for good UX with reasoning models, but it has overhead. Each chunk carries SSE framing and TLS record padding. For a model that emits 10k reasoning tokens, that overhead is negligible relative to compute. For a model that emits nothing for 20 seconds then dumps a block, streaming gives no TTFT benefit.

Polling a non-streaming endpoint forces you to wait for full completion, so you cannot measure TTFT at all. Do not use polling if your goal is to characterize time to first token reasoning models.

If your provider supports incremental reasoning streaming, surface those tokens in a collapsible “thinking” pane. That converts dead time into perceived progress.

The hidden cost of reasoning length

Reasoning models allocate tokens to thinking dynamically. A simple query might trigger a short hidden phase; a hard one triggers a long one. Your TTFT p95 should be measured on the hard set, because that is what breaks your UI.

Easy prompts may yield TTFT under a few seconds; hard prompts can exceed ten seconds, with tails much longer. If you sized a loading spinner based on easy-case averages, users hit timeouts on the hard cases.

Architectural implications

Engineers building on these models need to design for the worst observed TTFT, not the median. Use asynchronous SSE consumption and render a deterministic “reasoning initiated” state immediately.

If you sit behind an inference gateway, automatic fallback can mask a degraded provider but may reset the reasoning context. That tradeoff is acceptable when TTFT exceeds your SLA, but measure fallback cost explicitly.

Set client-side timeouts at the stream level: if no chunk arrives in 60 s, show a retry, but do not cancel the server job if it supports resumption.

Decisive takeaway

Stop publishing raw “TTFT” for reasoning models as a single number. Measure and report two metrics: time to first streamed reasoning character (if available) and time to first final-answer token. Label whether the run used prefix caching. Sample hard prompts.

Time to first token reasoning models is a function of provider streaming policy and problem difficulty, not a fixed hardware attribute. Build your UI to tolerate a 30-second silent phase, and you will ship a product that feels responsive even when the model is thinking hard.