What is time to first token? It is the wall-clock duration between when a client sends a completed prompt to an LLM inference endpoint and when the first generated token arrives at the client, before any subsequent tokens stream in. For interactive applications, this latency defines whether a user perceives the system as responsive or broken.
What Is Time to First Token, Precisely?
To expand on what is time to first token, we separate it from total generation time. The metric starts at the moment your HTTP request with the full prompt payload hits the server’s load balancer and ends when the first token of the completion is serialized and sent back over the wire. It does not include the time to generate the remaining tokens, nor does it include client-side rendering.
In streaming APIs, you see this as the gap before the first data: chunk. In non-streaming APIs, TTFT is effectively the entire response time because you get nothing until the full sequence is ready—but that conflates queuing, prefill, and decode. Always measure TTFT with streaming to isolate it.
How TTFT Is Measured
You need a clock on the client and a streaming response. Record t0 immediately before sending the request. As soon as you parse the first token from the stream, record t1. The difference is your TTFT.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
t0 = time.perf_counter()
stream = client.chat.completions.create(
model="mistral-7b",
messages=[{"role": "user", "content": "Explain Raft consensus."}],
stream=True,
)
first_token = None
for chunk in stream:
if chunk.choices[0].delta.content:
first_token = chunk.choices[0].delta.content
break
t1 = time.perf_counter()
print(f"TTFT: {t1 - t0:.3f}s")This ignores network jitter if your client is far from the region. For benchmarking, run from the same cloud region as the endpoint. A typical first chunk looks like:
{
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"model": "mistral-7b",
"choices": [
{
"delta": {"role": "assistant", "content": "R"},
"index": 0,
"finish_reason": null
}
]
}The arrival of that JSON object marks the end of the TTFT window.
What Happens Under the Hood
A request spends TTFT in several stages:
Queue and Scheduling
The gateway or inference server accepts the connection and places the request in a scheduler queue. Under load, this is where tail latency explodes. A single expensive prompt can block smaller ones if the scheduler is naive.
Prefill (Prompt Processing)
The model computes the key-value cache for the entire input prompt. This is compute-bound and scales with prompt length and model parameters. Long system prompts dominate TTFT here. Transformer attention is roughly quadratic in sequence length without sparse or fused kernels, so a 32k-token context costs far more prefill than a 2k one.
First Decode Step
The model samples the first token from the logits. This step is similar to every subsequent decode but only happens after prefill completes.
Network Egress
The token is serialized, compressed if using websockets, and sent. Usually minor compared to prefill unless the connection is terrible.
Why TTFT Matters More Than You Think
Users judge responsiveness at the sub-second level. If your chatbot sits silent for three seconds before showing anything, abandonment spikes even if the full answer arrives quickly. In agentic loops, TTFT compounds: a planner that calls a tool, waits for LLM, then calls another, multiplies latency by the number of steps.
For synchronous backend tasks—batch summarization—TTFT is irrelevant; total throughput matters. But for any UI, voice assistant, or interactive coding copilot, TTFT is the prime metric.
A Concrete Example: Streaming a Coding Assistant
Suppose you build a VS Code extension that suggests a function. The user triggers completion with a 200-line context window. You stream from a 13B model.
If TTFT is 800 ms, the user sees the first characters before they context-switch. If TTFT is 4 s because the provider queued the request behind a 32k-token batch, the extension feels broken. The fix is not a bigger model; it’s routing to a less loaded replica or using a model with faster prefill.
const t0 = performance.now();
const response = await fetch("https://api.example.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "codellama-13b",
messages: [{ role: "user", content: context }],
stream: true,
}),
});
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let first = true;
while (true) {
const { done, value } = await reader.read();
if (done) break;
if (first) {
console.log(`TTFT: ${performance.now() - t0} ms`);
first = false;
}
// handle chunk
}TTFT vs. Time to Last Token
Total interaction latency is TTFT + (output_tokens / decode_tokens_per_second). A model with low TTFT but slow decode can still feel sluggish on long answers. Conversely, a high TTFT followed by rapid streaming is acceptable for short outputs but fatal for perceived responsiveness on the first character.
Understanding what is time to first token helps you set service-level objectives that reflect user experience rather than backend convenience.
Common Misconceptions About TTFT
“TTFT Is the Same as Latency”
No. Latency often means time-to-complete. TTFT is strictly the first token. A model can have great TTFT but terrible tokens-per-second, producing a slow overall feel after the start.
“Smaller Models Always Win”
Not necessarily. A 7B model on overloaded hardware loses to a 70B on a dedicated GPU with optimized prefill. TTFT depends on deployment, not just parameter count.
“Streaming Eliminates the Problem”
Streaming moves the pain from silent wait to incremental wait, but the initial gap remains. If TTFT is high, users still see a frozen cursor.
“You Can Cache Your Way Out”
Prompt caching helps prefill if the prefix is reused. But cache misses—new system prompts, dynamic RAG context—pay full price. And cache lookup itself adds a tiny overhead.
“It’s the Provider’s Problem Alone”
Client-side timeouts, connection reuse, and DNS resolution contribute. A cold TLS handshake can add 100–200 ms before a single byte is sent.
TTFT and Routing Decisions
When you front multiple providers, TTFT becomes a routing signal. An inference gateway like n4n.ai can apply automatic fallback when a provider is rate-limited or degraded, which prevents a single vendor’s TTFT blowup from reaching your users. Without that, your code must implement retries and timeout budgets manually.
Even with fallback, honor provider cache-control hints. If a provider signals a cached prefix, the prefill cost drops and TTFT improves. Forwarding those hints is a routing directive that pays off.
Practical Levers to Reduce TTFT
- Trim prompts. Every token in the context is prefill work. Use concise system prompts.
- Use prompt caching for static prefixes (boilerplate instructions, few-shot examples).
- Choose regions close to your users or run inference in same zone as app servers.
- Set concurrency limits on clients to avoid self-induced queueing.
- Select models by prefill speed for latency-sensitive paths; reserve heavy models for offline.
- Pre-warm connections with HTTP keep-alive to avoid TCP/TLS handshakes on every call.
Capacity Planning With TTFT Targets
Set an SLO such as p95 TTFT < 1s for chat surfaces. Size GPU memory bandwidth and batch sizes to meet that under expected load. Monitor TTFT per model and per route; a regression in prefill kernel version can silently double it.
Run continuous probes from production-like locations. Plot p50, p95, p99—the tail is where UX dies. A p99 TTFT of 5s will haunt you even if the median is 300 ms.
TTFT in Non-Streaming Endpoints
If you must use a non-streaming API, you cannot measure TTFT directly; you only get total time. Some gateways emit an early header like x-first-token-ms for diagnostic purposes, but that is non-standard. Prefer streaming for any latency-sensitive path.
If you internalize what is time to first token as the moment your system earns the user’s patience, you will architect for it deliberately.