n4nAI
Open console →
Benchmarks & performanceAnalysisTime-to-First-Token Benchmarks

Time-to-first-token benchmark across 12 LLM providers

A time to first token benchmark providers analysis reveals why raw latency numbers mislead. Learn methodology, tradeoffs, and how to measure TTFT correctly.

n4n Team4 min read952 words

Audio narration

Coming soon — every post will get a voice note here.

A credible time to first token benchmark providers comparison is useless if it treats all models as interchangeable. TTFT depends on model architecture, prompt size, region, and whether the provider caches your prefix—variables that public leaderboards flatten into a single bar chart.

The thesis: TTFT is a distribution, not a number

Optimize for the tail, not the median. A provider that posts a 250ms median but a 4s p99 will wreck your interactive UX the moment you scale. The only time to first token benchmark providers exercise worth running is one that captures percentiles under your real concurrency and prompt shape.

Most published benchmarks report a single average from a warm region with a 50-token prompt. That hides the two things that matter in production: cold prefix compilation and queueing under load. If you ship a chat app, your users feel the p99, not the marketing p50.

Building a real measurement harness

Control four variables before you compare anything:

  1. Model parameter class (7B, 13B, 70B, frontier).
  2. Prompt token count (use a fixed 128-token and a 2k-token variant).
  3. Client region (ping from the same AZ you deploy in).
  4. Concurrency (1, 10, 50 simultaneous requests).

Stream the response and timestamp the first delta. Here is a minimal Python harness using the OpenAI SDK against any compatible endpoint:

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.your-gateway.com/v1", api_key="sk-...")

def measure_ttft(model, prompt):
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            return (time.perf_counter() - start) * 1000
    return None

ttft_ms = measure_ttft("mixtral-8x7b", "Explain TTFT in one sentence." * 10)
print(f"TTFT: {ttft_ms:.1f}ms")

For concurrency, wrap the call in an asyncio loop and collect samples:

import asyncio
from openai import AsyncOpenAI

aclient = AsyncOpenAI(base_url="https://api.your-gateway.com/v1", api_key="sk-...")

async def measure_one(model, prompt):
    start = time.perf_counter()
    stream = await aclient.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}], stream=True
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            return (time.perf_counter() - start) * 1000
    return None

async def run_bench(model, prompt, n):
    return await asyncio.gather(*[measure_one(model, prompt) for _ in range(n)])

samples = asyncio.run(run_bench("llama-3-70b", "Summarize: " + "x"*2000, 50))
print(f"p99: {sorted(samples)[-1]:.1f}ms")

Store raw latencies, not averages. Compute p50, p90, p99 with statistics.quantiles or numpy.

Why prompt length breaks naive rankings

A provider with a fast tokenizer and aggressive prefix caching can look slow at 128 tokens and win decisively at 2k. The time to first token benchmark providers matrix must include at least two prompt sizes or you will misallocate traffic.

The 12 providers and their shapes

A representative set of 12 providers today includes OpenAI, Anthropic, Google, Mistral, Together, Fireworks, Groq, Cerebras, Replicate, Perplexity, DeepInfra, and a self-hosted Ollama node. Without fabricating numbers, they split into three behavioral clusters:

Hyperscaler APIs

OpenAI, Anthropic, Google, Mistral. They run massive homogeneous fleets. TTFT is predictable but not always lowest because they prioritize fairness under huge multi-tenant load. Caching is available but gated behind specific headers.

GPU aggregators and inference specialists

Together, Fireworks, Groq, Cerebras, DeepInfra. These often colocate single-model replicas and can return first token in sub-300ms for small models due to optimized kernels. Their p99 suffers when a replica is evicted or a burst hits.

Open-weight hosts and self-host

Replicate, Perplexity (partially), Ollama. Median TTFT varies by region; tail latency is brutal because they overcommit GPUs or run on consumer hardware.

A time to first token benchmark providers run will show cluster 2 winning medians, cluster 1 winning tails, and cluster 3 being unpredictable. The moment you add concurrency, the order reshuffles.

Prompt caching: the silent variable

If your system sends a long system prompt on every request, TTFT without caching measures the wrong thing. Anthropic’s cache_control and similar hints let the provider skip recomputation:

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": "You are a strict JSON formatter. ... 2000 tokens of schema ...",
      "cache_control": {"type": "ephemeral"}
    },
    {"role": "user", "content": "Generate an object."}
  ]
}

A provider that honors this can drop TTFT from seconds to hundreds of milliseconds on the second call. Any time to first token benchmark providers comparison that ignores cache state is measuring cold starts only.

Gateways should forward these hints untouched. If you route through a layer that strips cache_control, you artificially inflate TTFT for every downstream provider that would have cached.

Concurrency and the tail: why p99 eats your UX

Fire a single request and every provider looks fine. Fire 50 and watch queueing. We model TTFT under load as:

observed_ttft = base_compute + queue_wait + network_rtt

queue_wait is zero at concurrency 1 and dominates at concurrency 50. The providers with dedicated autoscaling win here; those with static replicas fall off a cliff.

A decisive time to first token benchmark providers result is the p99 at your target QPS, not the p50 at idle. If a provider’s p99 at 10 QPS exceeds 2s, it is unfit for interactive use regardless of its median.

Regional placement and network RTT

Measuring from us-east-1 to a provider’s eu-central-1 endpoint adds 80–120ms of irreducible RTT before any compute. Many benchmarks omit this, then wonder why numbers differ from a colleague’s run. Pin your client and server regions, or the time to first token benchmark providers data becomes non-comparable.

Gateway abstraction as a latency smoother

A gateway that fronts multiple providers converts a single worst-case p99 into a blended distribution. n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and triggers automatic fallback when a provider is rate-limited or degraded. If you measure TTFT against that endpoint with routing left to the gateway, you get the best available median and a trimmed tail because failed or slow paths reroute.

That is not cheating the benchmark; it reflects how you should ship. You rarely care which GPU served the token.

Tradeoffs of the gateway approach

Abstraction costs visibility. When you pin a model through a gateway, you depend on its health checks. n4n.ai honors client routing directives and forwards provider cache-control hints, so you can force a specific backend if its TTFT suits your workload, but you still inherit the gateway’s retry overhead (typically 10–30ms).

You also face inconsistent streaming chunk sizes across providers, which can make TTFT measurement noisy if you define “first token” as “first meaningful content” versus “first raw chunk”. Standardize on the first non-empty delta.content and discard chunking differences.

Decisive takeaway

Run your own time to first token benchmark providers test with the three variables above, capture p99 at production concurrency, and treat any provider that cannot show cached and uncached numbers as incomplete. Then put a fallback-capable gateway in front so a single provider’s bad minute does not become your outage. The fastest median is a vanity metric; the shortest tail under load is the only number that ships.

Tagstime-to-first-tokenprovider-comparisonbenchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceClaude Opus 4.5 time-to-first-token across providersClaude Opus 4.5 time to first token depends on provider infrastructure, not just the model. We analyze cross-provider TTFT tradeoffs and routing tactics.Benchmarks & performanceGPT-5 time-to-first-token: cold start vs warm cacheAnalysis of GPT-5 time to first token cold start versus warm cache, breaking down prefill latency and cache strategies for production LLM apps.Benchmarks & performanceHow streaming affects time-to-first-token benchmarksPractical guide to measuring streaming time to first token: client timing, network buffering, fallback tradeoffs, and LLM benchmark pitfalls.Benchmarks & performanceLowest time-to-first-token models ranked for July 2026Ranked list of the lowest time-to-first-token models for July 2026, with real-world engineering context on latency measurement and infrastructure tradeoffs.Benchmarks & performanceMeasuring time to first token in production API callsA practical how-to for measuring time to first token in production LLM API calls: instrumentation, sampling, and pitfalls for accurate TTFT benchmarks.Benchmarks & performanceTime-to-first-token benchmark for reasoning modelsAnalyze why time to first token reasoning models benchmarks mislead engineers, and how to measure latency correctly for hidden inference workloads.

All time-to-first-token benchmarks posts →