Time to first token vs tokens per second are two distinct latency metrics for LLM inference: the former measures the wall-clock delay before the first generated token arrives, while the latter measures the sustained generation speed after that initial token. Confusing the two leads to broken UX assumptions and misleading benchmark comparisons. Both are expressed in milliseconds and tokens/second respectively, but they answer different questions about where time goes in a request.
What each metric captures
TTFT (time to first token) is the interval from when the client sends a complete request to when it receives the first piece of generated content. This includes connection setup, request routing, queue wait, prompt prefill, and the first decode step.
TPS (tokens per second) is the rate of token production during the streaming phase. You compute it by dividing the number of tokens generated after the first by the elapsed time between the first token and the last.
The distinction between time to first token vs tokens per second matters because they are governed by different parts of the serving stack. Prefill compute and scheduling determine TTFT; memory bandwidth and batching efficiency determine TPS. A system can have excellent TPS and terrible TTFT, or vice versa, depending on configuration.
How measurement works in practice
Client-side timing
Always measure with streaming enabled. A non-streaming request hides TTFT entirely and only gives you total latency. Use a monotonic clock, and subtract only the time spent in your own network hop if you control the client location.
import time
import openai
client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
start = time.perf_counter()
first_token_ts = None
token_count = 0
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain quicksort"}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
if first_token_ts is None:
first_token_ts = time.perf_counter()
ttft = first_token_ts - start
token_count += 1
end = time.perf_counter()
gen_time = end - first_token_ts
tps = (token_count - 1) / gen_time if gen_time > 0 else 0
print(f"TTFT: {ttft*1000:.0f}ms, TPS: {tps:.1f}")This snippet separates the two phases explicitly. Note that token_count - 1 excludes the first token from the TPS denominator, matching most provider definitions. If you include the first token, you couple prefill cost into the generation rate and obscure the decode bottleneck.
Server-side factors
On the server, TTFT is dominated by prefill: the transformer forward pass over the input tokens to build the KV cache. A 4k-token prompt costs far more prefill compute than a 40-token prompt, even if the generated output is identical. Prefill is quadratic in sequence length for naive attention, though flash attention makes it closer to linear with a large constant. Regardless, a 32k context window takes noticeably longer to prefill than a 2k one.
TPS is dominated by decode-step latency. Each step processes one (or a small group of) token(s) against the KV cache. Memory-bound GPUs peak at a certain tokens/sec per sequence; continuous batching improves utilization but can throttle individual sequence TPS under load. Once decoding, TPS is largely independent of prompt length because KV cache reads are constant per step, but pageable KV cache eviction under memory pressure can cause sporadic drops.
Why the distinction matters for real systems
A chat interface feels broken if TTFT exceeds ~500ms, regardless of how fast the rest streams. Users perceive the model as “thinking” too long. Conversely, a batch job that summarizes 10k documents cares almost exclusively about TPS, because nobody is waiting on the first byte.
When you sit behind an inference gateway such as n4n.ai that performs automatic fallback on provider degradation, the TTFT distribution widens because a retry may hit a colder instance, but the streaming TPS after the first token stays tied to the model’s decode speed. Per-token usage metering still lets you reconcile billed tokens regardless of which backend served the request.
Consider a coding copilot: the user expects a suggestion within a heartbeat, so TTFT is the product metric. A RAG pipeline embedding 1M chunks overnight cares about aggregate TPS across thousands of sequences. The same model can rank first in one and last in the other. If you optimize only one metric, you will misfire. Tuning batch sizes raises TPS but often worsens TTFT due to queue contention. Adding speculative decoding cuts TTFT but may not move TPS much for long outputs.
A concrete example
Assume a 100-token prompt and a 200-token completion.
Provider A: TTFT 200ms, TPS 50. Total time = 0.2s + (200 / 50)s = 4.2s.
Provider B: TTFT 800ms, TPS 80. Total time = 0.8s + (200 / 80)s = 3.3s.
Provider B is faster end-to-end despite a 4x worse TTFT. For an interactive chatbot, Provider A feels snappier at the start; for a nightly ETL pipeline, Provider B wins.
Now shrink the output to 20 tokens:
Provider A: 0.2 + 20/50 = 0.6s. Provider B: 0.8 + 20/80 = 1.05s.
Same providers, opposite winner. That is why time to first token vs tokens per second must be reported separately, with output length called out.
Common misconceptions
“TPS is the only speed number that matters.” Wrong. If your app streams to a UI, TTFT is the user’s first impression. A 100 TPS stream that takes 2s to start feels slower than a 40 TPS stream that starts in 150ms.
“TTFT is just network latency.” No. On a 1k-token prompt, prefill alone can take hundreds of milliseconds on commodity GPUs. Network is often <30ms intra-region.
“Reported TPS includes the first token.” Most measurements start the clock at first token receipt. Including prefill in the generation rate artificially deflates it.
“Short-output benchmarks are representative.” A 16-token completion is 90% TTFT and 10% generation. Published “average TPS” from such tests is meaningless for long-form generation.
“Time to first token vs tokens per second are interchangeable under load.” Under congestion, TTFT balloons while TPS stays flat until the scheduler admits your request. Treating them as one “latency” metric hides the queueing signature.
“Higher TPS always means lower cost.” Not necessarily. A provider charging per token at higher TPS but with worse queueing may cost the same total because you wait for admission, and you may need larger timeouts.
Measuring correctly
- Use production-like prompt lengths. Mirror your real input distribution, not a 10-token hello.
- Run at least 50 trials and report p50/p90/p99. A single sample is noise.
- Separate phases in logs:
request_sent,first_token,stream_done. - Account for tokenization. Count tokens via the model’s tokenizer, not string length. A “token” is not a word; mismatch across models invalidates TPS comparisons.
- If you use a gateway, honor its cache-control hints. A cached prefill can cut TTFT by an order of magnitude; forward
cache_controlto get repeat-call savings and measure the delta.
When comparing models or providers, publish both numbers with the output token count. A table that shows only “latency” is either hiding something or misunderstanding the system.
Engineers who internalize the split between time to first token vs tokens per second build better interfaces and choose backends based on actual workload shape rather than marketing sheets.