Time to first token: what it measures and why it matters

Time to first token (TTFT) is the elapsed wall-clock duration between sending a completed prompt to an LLM inference service and receiving the first generated token in the response stream. It captures the fixed upfront cost of request handling—network round-trip, queueing, prompt preprocessing, and the first forward pass—before token generation settles into a steady-state tokens-per-second rhythm. Unlike inter-token latency, time to first token isolates the perceived “thinking” delay that users feel before a response starts appearing.

How time to first token is measured

The measurement sounds trivial until you account for where the clock actually starts. If you start the timer before you finish uploading a large prompt body, you are measuring request serialization, not server thinking. If you stop it after parsing the HTTP headers but before the first delta chunk, you are ignoring proxy buffering.

The clock starts and stops where?

Start the clock the moment the client hands a complete request to the transport layer (typically after send() on the socket or the return from the HTTP library’s request call with stream=True). Stop it the moment you decode the first non-empty choices[0].delta.content from a chunked response. Everything in between is TTFT.

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft = time.perf_counter() - start
        break

Breaking down the latency budget

Time to first token is not a single operation. It is a sum:

• Network egress: client → load balancer → inference worker.
• Auth and routing: API key validation, model lookup, provider selection.
• Queueing: the request waits for a free batch slot. Under load this dominates.
• Prompt encoding: tokenization, KV-cache allocation, prefix cache lookup.
• Prefill: the forward pass over the full prompt. This scales with prompt length.
• Sampling the first token: logits → sampling → detokenization to a streamable delta.

A single OpenAI-compatible endpoint such as n4n.ai exposes 240+ models behind the same /v1/chat/completions contract, so the snippet above works unchanged regardless of which backend you route to. The prefill cost differs wildly between a 70B model and a 7B model, but the measurement method does not.

Why time to first token matters in production

Engineers obsess over tokens-per-second, but users judge systems by TTFT. A chatbot that emits the first word in 150ms and then streams slowly feels faster than one that takes 3 seconds before anything moves, even if the latter finishes first.

Perceived responsiveness

Human attention decays in hundreds of milliseconds. If the time to first token exceeds roughly a second, users perceive the system as broken and retry, which multiplies load. Streaming UIs exist precisely to spend the TTFT budget on showing a typing indicator or skeleton, but the indicator itself does not reduce the metric.

Streaming UI and partial rendering

In a RAG pipeline, you often render the citation bar or a “thinking” state while waiting. The moment the first token arrives, you swap to real content. Your frontend code should key off the same event that stops the TTFT clock:

const reader = response.body.getReader();
const decoder = new TextDecoder();
let firstTokenAt: number | null = null;
const start = performance.now();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const text = decoder.decode(value);
  if (!firstTokenAt && text.includes('"content":"')) {
    firstTokenAt = performance.now();
    console.log(`TTFT: ${firstTokenAt - start}ms`);
  }
}

Capacity and cost signals

TTFT inflates when batch utilization is high. Watching TTFT per model lets you distinguish “the model is slow” from “we are under-provisioned.” If prefill is the bottleneck, adding more replicas helps. If inter-token latency is fine but TTFT is spiking, you are queueing, not computing.

A concrete measurement example

Suppose you serve a coding assistant. The prompt is a 2,000-token system prompt plus a 200-token user query. You call a gateway with automatic fallback.

Client-side Python snippet

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")

prompt = "Refactor this function to use async generators."

start = time.perf_counter()
stream = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": prompt}],
    stream=True,
)
ttft = None
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        ttft = time.perf_counter() - start
        print(f"Time to first token: {ttft*1000:.1f} ms")
        break
# consume rest to avoid broken pipe
for _ in stream:
    pass

This prints the time to first token for that specific request. Run it 100 times across the day and you get a distribution, not a number.

Server-side variables you don’t control

Your client measures TTFT, but the server may have:

• Cold KV-cache vs warm prefix cache (cache-control hints forwarded by the gateway).
• Batch coalescing where your request waits for others.
• Provider degradation triggering fallback to a secondary region.

These are invisible to the client except as variance in the metric. That is why you track percentiles, not averages.

Common misconceptions about time to first token

It’s just network ping

Network latency is usually a small fraction of TTFT for any non-trivial prompt. Prefill of a long context on a large model can take seconds; a cross-region ping is tens of milliseconds. Blaming the network is a rookie mistake.

Lower TTFT means a better model

A smaller model often has lower time to first token because prefill is cheaper. That does not make it “better” at reasoning. Conversely, a model with aggressive speculative decoding may show great TTFT but worse final quality. Optimize the metric for the UX, not as a proxy for capability.

TTFT is stable per model

It is not. The same model behind the same API can show 300ms TTFT at 2am and 4s at noon because of queue depth. Prompt length changes prefill linearly-ish. Prefix cache hits can drop TTFT by an order of magnitude. Treat TTFT as a distribution keyed by (model, prompt_length, load, cache_state).

You must scale GPUs to fix it

Sometimes the win is algorithmic. Continuous batching, prefix caching, and rejecting oversized prompts at the edge reduce TTFT without new hardware. Honoring provider cache-control hints from the client lets the gateway skip recomputation. Routing directives that pin a request to a warm replica cut cold-start penalties. Those are engineering levers, not credit card levers.

Monitoring TTFT without drowning in data

Emit TTFT as a histogram metric tagged by model and route. Alert on p95, not mean. When p95 time to first token doubles, page someone—that is the earliest signal of saturation or a degraded upstream provider. The metric is cheap to collect (one timestamp per stream) and pays for itself the first time it catches a silent fallback storm.