n4nAI
Open console →
Benchmarks & performanceGuideTime-to-First-Token Benchmarks

Time to first token vs total latency: what to measure

Engineer's guide to measuring time to first token vs total latency: instrument both, avoid pitfalls, and optimize LLM app responsiveness.

n4n Team5 min read1,143 words

Audio narration

Coming soon — every post will get a voice note here.

Optimizing a language model endpoint starts with knowing which clock you care about. The gap between time to first token vs total latency determines whether your users feel a system is snappy or merely complete quickly, and the two numbers respond to different knobs. Treat them as separate signals from the first day of benchmarking.

Define what your application actually needs

You cannot optimize what you haven’t scoped. A conversational assistant and a nightly document classifier have opposite latency profiles. Write down the interaction pattern before writing a benchmark script.

Interactive chat

Low TTFT dominates. Users perceive lag above ~300–500ms before the first character renders. Total latency matters less if the stream paints smoothly and the inter-token gap stays under ~50ms. A voice agent tolerates higher total latency if TTFT is under 200ms because users attribute silence to the model “thinking.”

Batch and offline

Throughput per dollar wins. Total latency across 10k documents is the metric; TTFT is irrelevant when you fire requests asynchronously and collect results later. A summarization cron job has no human watching the clock.

Hybrid RAG

First token includes retrieval, reranking, and prefill. Separate those stages or you will blame the model for your vector database. Measure the span from “query received” to “first model token” as one TTFT, but log retrieval time as a sub-span so you can attribute regressions.

Instrument both metrics at the client boundary

Server-side logs lie about user-perceived delay. Measure from the process that sends the request to the process that renders the token. Run this from the same network context as production—a laptop in a different continent measures VPN overhead, not model speed.

import openai, time

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="YOUR_KEY")
start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain quicksort in one paragraph"}],
    stream=True,
)
first_ts = None
tokens = 0
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        if first_ts is None:
            first_ts = time.perf_counter()
        tokens += 1
end = time.perf_counter()

ttft_ms = (first_ts - start) * 1000
total_ms = (end - start) * 1000
print(f"TTFT: {ttft_ms:.1f}ms | Total: {total_ms:.1f}ms | Tokens: {tokens}")

Export as histograms so you can compute percentiles later:

from prometheus_client import Histogram
TTFT = Histogram('llm_ttft_seconds', 'Time to first token')
TOTAL = Histogram('llm_total_latency_seconds', 'Total request latency')
# after the loop
TTFT.observe(ttft_ms / 1000)
TOTAL.observe(total_ms / 1000)

Record p50, p95, p99. A single median hides the tail that churns users. If you only alert on averages, a p99 TTFT blowup from a degraded provider will slip through.

Account for streaming chunk sizes and network effects

Providers package tokens differently. One may emit per-token SSE frames; another buffers 20 tokens per frame. Your TTFT measurement stays correct, but inter-token cadence and total transfer time shift. Do not compare “tokens per second” across providers without normalizing chunking.

Proxy buffering is a silent killer. A default nginx config can hold SSE frames until a buffer fills, adding 100–200ms to TTFT with zero model change. Set proxy_buffering off; for streaming routes.

TCP slow start and TLS handshake count in TTFT if you open a new connection per request. Reuse connections—the OpenAI SDK pools them when you keep the client instance alive. HTTP/2 multiplexing helps under concurrency but does not change single-stream TTFT.

Measure decoded tokens, not frames

Count delta.content length, not chunks. Some chunks carry role or finish_reason with no content. A frame with empty content must not reset your first-token timer.

Separate provider latency from gateway overhead

When you sit an inference gateway in front of multiple providers, the abstraction changes the distribution. A gateway such as n4n.ai performs automatic fallback when a provider is rate-limited or degraded, which stretches the p99 of time to first token vs total latency asymmetrically: TTFT spikes because the retry picks a slower backend, while total latency may degrade less if the fallback streams faster overall. The same gateway honors client routing directives and forwards provider cache-control hints, so a cached prefix collapses TTFT without altering generation speed.

Benchmark the raw provider endpoint and the gateway path separately. Subtract the difference to size the overhead. If the gateway adds more than 30ms p50 TTFT, inspect your region placement, not the model. Per-token usage metering on the gateway lets you confirm you are billed for the same tokens you counted client-side—a sanity check that your measurement loop isn’t dropping streamed chunks.

Run controlled benchmarks with representative prompts

Synthetic “hello world” prompts produce misleadingly low TTFT because prefill is trivial. Use real prompts from your logs, including long system messages and few-shot examples.

Methodology

  1. Sample 200 production prompts across length percentiles (p10, p50, p90).
  2. Run 50 warmups to populate provider caches and JIT compiles.
  3. Execute 500 measurements at concurrency 1 for latency, then at target concurrency for throughput.
  4. Tag each run with model, route, and cache-hit header.

Aggregate as JSON:

{
  "model": "gpt-4o-mini",
  "cache_hit": true,
  "ttft_p50_ms": 142,
  "ttft_p95_ms": 310,
  "total_p50_ms": 1800,
  "total_p95_ms": 2400,
  "tokens": 320
}

Compare time to first token vs total latency across cache states. A 5x TTFT drop on cache hit is normal and should not be confused with model speedup. Under concurrency, total latency grows with queue depth; TTFT grows with both queue and prefill contention.

Common pitfalls when comparing time to first token vs total latency

  • Averaging over mixed traffic. Support bots and code gen have different prompt lengths. Slice by route or you will report a number that describes no real user.
  • Ignoring cold prefill. First request after deploy pays load/compile cost. Exclude or label it, or your baseline is unusable.
  • Using wall-clock on shared hosts. Noisy neighbor on a cheap VM skews total latency. Pin the client to a dedicated core.
  • Treating TTFT as pure model property. It includes network, gateway, and queue time. Attribute each with spans before blaming the provider.
  • Assuming linear scaling. Doubling batch size rarely doubles total latency, but TTFT often climbs sharply because prefill is serialized behind the batch.
  • Trusting provider dashboard only. Their start timestamp may be after request receipt. Your client timestamp is the source of truth for UX.

When you report numbers, state concurrency and client location. “TTFT 200ms” without context is fiction.

Map metrics to user-perceived experience

TTFT under 300ms feels instant in a chat UI. 300–800ms feels responsive but mechanical. Above 1s, users switch tabs. Total latency per output token sets the “typing speed” perception; a steady 30 tokens/sec feels like a fast human, while 10 tokens/sec feels like a slow terminal.

If your product is a copilot, optimize TTFT first, then inter-token latency. If it is a bulk classifier, optimize total latency per 1k tokens and ignore TTFT entirely. The correct weighting is a product decision, not a default.

Tradeoffs in optimization: where to spend engineering effort

Lowering TTFT usually means attacking prefill: prompt caching, smaller models for first pass, or speculative decoding. Lowering total latency means attacking decode: batching, quantization, more GPUs.

These conflict. Maximal batching improves total throughput but queues requests, pushing TTFT up. Speculative decoding can cut total latency but adds prefill compute that may raise TTFT under load.

A practical order:

  1. Enable provider cache-control headers; verify hits with response extensions.
  2. Move gateway and client to same region as model.
  3. Switch high-TTFT routes to a smaller model with a larger model for refinement.
  4. Only then tune serving params (max batch, flash attention, quantization).

Measure time to first token vs total latency after each step. If a change improves one by 20% and worsens the other by 50%, reject it unless the use case justifies. Latency budgets are zero-sum; spend them where the user looks.

Tagstime-to-first-tokenlatencybenchmark-methodology

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceHow streaming affects time-to-first-token benchmarksPractical guide to measuring streaming time to first token: client timing, network buffering, fallback tradeoffs, and LLM benchmark pitfalls.Benchmarks & performanceLowest time-to-first-token models ranked for July 2026Ranked list of the lowest time-to-first-token models for July 2026, with real-world engineering context on latency measurement and infrastructure tradeoffs.Benchmarks & performanceMeasuring time to first token in production API callsA practical how-to for measuring time to first token in production LLM API calls: instrumentation, sampling, and pitfalls for accurate TTFT benchmarks.Benchmarks & performanceTime-to-first-token benchmark for reasoning modelsAnalyze why time to first token reasoning models benchmarks mislead engineers, and how to measure latency correctly for hidden inference workloads.Benchmarks & performanceWhy time to first token spikes during peak hoursEngineering analysis of why time to first token peak hours degrades under load, covering queueing, batching, and provider limits with concrete fixes.Benchmarks & performanceWhat is time to first token and why it mattersTTFT measures LLM latency from request to first response token. Learn what is time to first token, why it matters, and how to measure it.

All time-to-first-token benchmarks posts →