Flagship LLM speed rankings from GPT-5 to Grok 4

The latest flagship LLM speed rankings matter because latency directly gates user experience and cost in production inference. We cut through marketing to compare GPT-5, Grok 4, and their closest competitors on the metrics that actually show up in your dashboards: time-to-first-token (TTFT), tokens per second under load, and tail latency at the 99th percentile.

1. GPT-5

OpenAI’s GPT-5 continues the trend of mixture-of-experts (MoE) routing first seen at scale with GPT-4-class models. From an inference standpoint, the model is served behind heavily optimized Triton ensembles with speculative decoding on the prefill path. In practice you see sub-300ms TTFT on small prompts from US regions, but throughput degrades nonlinearly when batch size exceeds 64 because expert imbalance forces load shedding on the least-utilized experts.

Engineers should treat GPT-5 as a low-latency frontend model when paired with function calling, but avoid stacking long system prompts without caching. The OpenAI-compatible chat endpoint honors cache_control on system blocks, which can cut prefill cost by 40–60% on repeated contexts. That matters more than raw token speed for multi-turn agent loops where the system prompt is static across thousands of calls.

If you measure it yourself, pin the model version and stream tokens so you capture true interactive latency rather than a single blocking response:

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
t0 = time.perf_counter()
resp = client.chat.completions.create(
    model="gpt-5",
    messages=[{"role":"user","content":"Summarize distributed inference."}],
    stream=True,
    stream_options={"include_usage": True}
)
for c in resp:
    pass
print(f"wall: {time.perf_counter()-t0:.2f}s")

The key takeaway: GPT-5 wins on interactive feel but punishes naive batching.

2. Grok 4

xAI built Grok 4 on a custom RDMA-connected XPU cluster, and the speed signature reflects that vertical integration. Single-stream TTFT is competitive with GPT-5, but the standout is sustained throughput on long generations: the scheduler favors continuous batching with large KV cache sharding across nodes. Under 2k-token prompts with 8k completion requests, Grok 4 holds roughly 1.4x the tokens/sec of comparable dense transformers.

The caveat is regional availability. If your traffic routes outside of US-east, you may hit fallback paths that add 100–200ms of network overhead before the first token. When you aggregate these behind one OpenAI-compatible endpoint like n4n.ai, you get automatic fallback when a provider is rate-limited, plus per-token metering that keeps the cost curve visible across regions without writing your own retry layer.

Use Grok 4 where you need high output volume per request—log summarization, code generation, synthetic data pipelines—and can tolerate slightly higher tail latency on cold starts. Its decode path is optimized for bulk, not for the sub-second chat loop where prefill dominates.

3. Claude 3.5 Opus

Anthropic’s Claude 3.5 Opus prioritizes reasoning consistency over raw speed. The architecture uses a smaller active parameter count per token than GPT-5’s MoE, which yields predictable latency but lower peak throughput. In load tests, TTFT hovers around 400–500ms for 1k context, and tokens/sec drops gently under concurrency rather than cliffing when the batch fills.

For engineers, the differentiator is the prompt caching lease. Claude’s cache_control epochs last hours, not minutes, making it ideal for static system prompts in agent loops. The flagship LLM speed rankings often undersell this because they measure cold calls; warm cache changes the equation entirely, turning a 500ms prefill into a 20ms cache hit.

If your product is a long-horizon agent that re-sends the same 2k-token instruction set on every turn, Claude 3.5 Opus will feel faster in aggregate than a theoretically quicker model that re-prefills every time. That is a routing decision, not a benchmark number.

4. Gemini 2.0 Ultra

Google’s Gemini 2.0 Ultra leverages TPU v5p pods with spatial partitioning. The result is exceptional prefill parallelism: multi-page document ingestion shows TTFT under 250ms even at 32k input tokens. However, decode throughput per stream is modest because TPU matrix units are prefill-biased and the all-reduce across slices adds fixed cost per step.

If your workload is retrieval-heavy with short answers, Gemini leads the flagship LLM speed rankings for prefill-bound tasks. For long autoregressive generation, schedule smaller batches to avoid inter-node stalls. You can express provider routing hints directly in the request payload when your gateway forwards them:

{
  "model": "gemini-2.0-ultra",
  "routing": { "prefer": "tpv5p-us-central" },
  "cache_control": { "type": "ephemeral", "ttl": "30m" }
}

The model respects client routing directives and forwards provider cache-control hints, so you can pin a fast zone without forking your client code.

5. Llama 4 Behemoth

Meta’s Llama 4 Behemoth is the only open-weight entry here, which means you control the serving stack. With vLLM and tensor parallelism of 8 on H100s, you can hit TTFT comparable to GPT-5 on 8k contexts, but only if you implement prefix caching and paged attention correctly. The speed ceiling is your hardware budget, not the model weights.

The tradeoff is operational. You own fallback, autoscaling, and quantization. FP8 cuts memory footprint by roughly 30% but adds 5–10ms per decode step on older GPUs. For teams already running Kubernetes, this is the most tunable entry in the flagship LLM speed rankings—you can trade latency for cost by adjusting gpu_memory_utilization and max_num_seqs in the engine config.

A minimal vLLM launch for Behemoth looks like:

vllm serve meta-llama/Behemoth-4 --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 --max-num-seqs 256 --enable-prefix-caching

Self-hosting shifts the speed conversation from model to middleware.

Synthesis

Speed is workload-shaped. No single model tops every column. Rank by your dominant cost pattern:

The flagship LLM speed rankings only become actionable when you map them to your own prompt length, concurrency, and region. Benchmark with production-shaped traffic, not synthetic 32-token prompts, and let the gateway handle fallback.