n4nAI
Open console →
Competitor comparisonsAnalysisInference speed benchmarks

Time to first token: comparing five LLM API gateways

Engineering analysis of time to first token LLM API comparison across five gateways, with architecture tradeoffs and a DIY latency measurement snippet.

n4n Team4 min read898 words

Audio narration

Coming soon — every post will get a voice note here.

Time to first token LLM API comparison is more about gateway architecture than model weights. After profiling five major inference gateways—OpenAI, Anthropic, Azure OpenAI, Groq, and OpenRouter—the decisive factor is how each handles request routing, cold starts, and prompt caching. Raw model speed matters, but the gateway layer often adds 50–500ms before a single token leaves the provider.

What TTFT actually measures

Time to first token (TTFT) is the wall-clock duration from when your client sends a request to when it receives the first streamed token. It includes DNS resolution, TLS handshake, authentication, request queuing, model warm-up, prompt preprocessing, and the first forward pass. It is not the same as inter-token latency or total generation time.

For interactive applications—chat, autocomplete, agent loops—TTFT dominates perceived responsiveness. A 2-second TTFT feels broken even if the subsequent tokens stream at 100 tok/s.

The five gateways under review

We compare these endpoints because they cover the spectrum of hosting models: first-party mono-model, hyperscaler, custom silicon, and aggregator.

OpenAI (api.openai.com)

The reference implementation. OpenAI serves its own models from a global fleet. TTFT is generally stable for small models (gpt-4o-mini) and climbs with context size on gpt-4 class models. No cross-provider routing; if the model is degraded, you get an error.

Anthropic (api.anthropic.com)

First-party hosting for Claude. Anthropic’s API uses a distinct message format (not OpenAI-compatible by default, though shims exist). TTFT scales with prompt length due to their attention implementation; public reports indicate competitive but not class-leading TTFT on 200K context windows.

Azure OpenAI

Same models as OpenAI but deployed in tenant-specific or shared instances. TTFT varies wildly by deployment type: provisioned throughput instances give predictable sub-second TTFT; shared quota can queue requests behind other tenants, spiking to multi-second.

Groq

Custom LPU hardware. Groq does not serve NVIDIA GPUs; their stack is built for token generation. For models they host (Llama 70B, Mixtral), TTFT is consistently the lowest among public gateways—often below 200ms even at moderate context. Tradeoff: limited model selection and no long-context variants.

OpenRouter

Aggregator that routes to multiple upstream providers (Together, Fireworks, Anthropic, etc.) behind one OpenAI-compatible API. TTFT depends entirely on which provider fills the request. You can pin a provider, but default routing optimizes for price/availability, not latency.

Architectural levers that move TTFT

Cold starts and container scheduling

Serverless inference (many aggregator routes) spins containers per model on demand. If a model hasn’t been called in minutes, the first request pays a cold-start tax of 300ms–2s. Dedicated deployments avoid this but cost more. Groq’s fixed-function fleet has near-zero cold start; Azure provisioned instances are warm; OpenAI and Anthropic keep large warm pools but still scale dynamically.

Prompt caching and pre-processing

Gateways that support prompt caching (Anthropic’s cache_control, OpenAI’s upcoming similar hints) can skip re-processing long system prompts. If your request hits a cache, TTFT drops sharply. Some gateways forward provider cache-control hints; for example, n4n.ai honors client routing directives and forwards provider cache-control hints, so a cached prefix on the origin propagates through the gateway. That removes a redundant pre-processing step.

Routing and fallback

Aggregators add a routing decision before the request reaches a model. Smart routing improves TTFT under degradation. A gateway with automatic fallback when a provider is rate-limited or degraded can shave seconds off worst-case TTFT by avoiding a queued 429 retry loop. Mono-provider gateways lack this escape hatch.

Measuring it yourself

Don’t trust vendor dashboards. Measure from your runtime. Here is a minimal Python snippet using the OpenAI client (works against any OpenAI-compatible endpoint):

import time
from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://api.openai.com/v1")

start = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain TTFT in one sentence."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft_ms = (time.perf_counter() - start) * 1000
        print(f"TTFT: {ttft_ms:.0f}ms")
        break

Run this from the same region as your production client. Repeat 100 times, drop the top and bottom 10 percentiles, and you have a real number. For aggregators, log the x-request-id and any provider header to see who actually served the token.

If you need raw HTTP timing, curl -w "@curl-format.txt" with a streaming POST gives similar data without SDK overhead.

Tradeoffs: speed vs flexibility

Groq wins on TTFT but locks you to three or four models. OpenAI and Anthropic give breadth and decent latency but no fallback if their status page goes red. Azure gives you control via provisioned instances if you can pay. OpenRouter gives you 100+ models but TTFT is a lottery unless you pin.

The hidden cost is client complexity. If you code against five gateways directly, you manage five auth schemes, five error shapes, and five latency profiles. An OpenAI-compatible aggregation layer collapses that to one client—but only if it doesn’t add a proxy hop that inflates TTFT.

A note on metering

Per-token usage metering matters when you compare gateways because a gateway that batches or pre-fetches may report different token counts. If your comparison includes cost, ensure you capture usage from the streamed response, not just the bill. Some gateways meter at the proxy; others pass through provider metering. Know which you are billed on.

Decisive takeaway

For a time to first token LLM API comparison, pick by workload:

  • Latency-critical, few models: Groq. Nothing else comes close for the models it hosts.
  • Predictable enterprise latency: Azure OpenAI with provisioned throughput.
  • Broad model access with acceptable TTFT: OpenAI or Anthropic first-party.
  • Maximum model choice, variable TTFT: OpenRouter with provider pinning.

If you need one endpoint that addresses 240+ models and degrades gracefully, a gateway with automatic fallback and cache forwarding will beat a raw aggregator on worst-case TTFT. But measure it from your own client before you commit—vendor TTFT numbers are marketing, your perf_counter is truth.

Tagslatencytime-to-first-tokenbenchmarkgateway

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Competitor comparisonsn4n.ai vs OpenRouter: inference latency benchmarkedA practitioner's head-to-head comparison of n4n.ai vs OpenRouter latency: gateway overhead, fallback behavior, cost, and ergonomics for production LLM systems.Competitor comparisonsStreaming latency in LLM APIs: what actually mattersDefines streaming latency LLM API: time to first token and inter-token delays, why they differ from batch latency, and how to measure what users feel.Competitor comparisonsWhy inference speed varies so much between LLM API providersWhy LLM API inference speed varies between providers: hardware, batching, quantization, and network factors explained with measurement code for engineers.Competitor comparisonsBenchmarking Mixtral 8x7B speed across inference providersA practical analysis of Mixtral 8x7B speed benchmark results across inference providers, covering measurement methods, tradeoffs, and what matters in production.Competitor comparisonsCerebras vs Groq vs SambaNova: speed benchmark comparedA practitioner's head-to-head comparison of Cerebras vs Groq vs SambaNova speed benchmarks: latency, throughput, cost, and ergonomics for LLM inference.Competitor comparisonsDeepSeek V3 inference speed: which API is fastestA practical DeepSeek V3 inference speed comparison across official API, OpenRouter, Fireworks, and Together, with measurement code and a clear verdict.

All inference speed benchmarks posts →