GPT-5 speed benchmark: latency, throughput, and cost

A GPT-5 speed benchmark published by a vendor will show you a carefully curated single-stream tokens-per-second number that hides the only metrics that break production: tail latency under concurrency and cost per corrected token. This analysis tears down what a real GPT-5 speed benchmark must measure, uses known behavior of transformer serving to set expectations, and gives you a decision framework for spending money on speed.

The thesis: speed is a distribution, not a point

Engineers default to asking “how fast is GPT-5?” as if it were a constant. It isn’t. The model’s inference speed is a function of batch size, prompt length, output length, and the provider’s fleet utilization at that millisecond. A GPT-5 speed benchmark that reports a median inter-token latency without the p99 is actively misleading.

The decisive factor for most applications is not the raw generation rate but the probability that a request lands in the slow tail when your traffic spikes. That tail is where retries, timeouts, and user frustration live.

Latency: where the seconds hide

Time to first token (TTFT)

TTFT measures the gap between sending the request and receiving the first generated token. It bundles network round trip, prompt processing (prefill), and scheduler queue time. For decoder-only models, prefill scales roughly linearly with prompt tokens; a 4k-token system prompt will cost you regardless of model generation speed.

A minimal measurement harness looks like this:

import openai, time, statistics

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")
# Substitute the real GPT-5 model id when available
def measure_ttft(model, prompt, runs=30):
    samples = []
    for _ in range(runs):
        t0 = time.perf_counter()
        stream = client.chat.completions.create(
            model=model, messages=[{"role":"user","content":prompt}], stream=True)
        for chunk in stream:
            if chunk.choices[0].delta.content:
                samples.append(time.perf_counter() - t0)
                break
    return statistics.median(samples), statistics.quantiles(samples, n=10)[-1]

med, p90 = measure_ttft("gpt-5", "Summarize the CAP theorem.")
print(f"median={med*1000:.0f}ms p90={p90*1000:.0f}ms")

Run that against a stable endpoint and you’ll see the p90 TTFT often doubles the median once the provider’s global queue warms up. A GPT-5 speed benchmark that only quotes the median is describing a quiet lab, not your 2 p.m. traffic.

Inter-token latency (ITL)

Once generation starts, ITL determines perceived fluency. Streaming UI masks a lot, but a 200ms/token model feels sluggish for code completion. ITL is bounded by the per-step compute of the model and the batch it shares with others. Larger flagships will not magically beat physics: each step still requires a full forward pass over the residual stream.

Throughput: concurrency breaks the naive math

Single-stream tokens/sec is a vanity metric. Production serves dozens to thousands of concurrent requests. Throughput per GPU scales with batch size until KV cache memory saturates. A GPT-5 speed benchmark must report aggregate tokens/sec across a sustained load test, not one curl.

Consider a simple load pattern:

# 50 concurrent requests, each asking for 256 tokens
for i in $(seq 1 50); do
  curl -s https://api.openai.com/v1/chat/completions \
    -H "Authorization: Bearer $KEY" \
    -d '{"model":"gpt-5","messages":[{"role":"user","content":"Write a SQL query for last 7 days active users"}],"max_tokens":256}' &
done
wait

If the provider autoscales aggressively, you’ll see near-linear throughput gains up to a point, then a cliff. That cliff is the real capacity limit. Architectures that assume infinite elastic throughput will page you.

Cost: the multiplier on every optimization

Speed without cost context is a hobby. Flagship model pricing typically scales with parameter count and demand. If GPT-5 costs 3x a prior flagship per token, shaving 100ms off TTFT by over-provisioning is a negative ROI for most batch jobs.

A concrete tradeoff: for a nightly ETL summarization of 10M tokens, a 20% slower but 40% cheaper model saves real money and nobody cares about latency. The GPT-5 speed benchmark matters most for interactive surfaces where user-perceived delay directly hits conversion.

Benchmark methodology traps

Prompt and output length skew

A benchmark using 32-token prompts and 8-token outputs measures scheduler overhead, not model speed. Real RAG prompts are long; real agent outputs are long. Always match your production distribution.

Cold starts and provisioned capacity

Some providers run flagships on dynamic fleets. The first request after a lull sees seconds of TTFT. A GPT-5 speed benchmark that warms up the endpoint for an hour before measuring is invalid for spiky workloads.

Routing and fallback noise

Most teams don’t call a single provider directly. They sit behind an OpenAI-compatible gateway. When running a clean GPT-5 speed benchmark through n4n.ai, set the routing directive to disable fallback so you measure the model, not the mesh:

{
  "model": "gpt-5",
  "stream": true,
  "route": { "prefer": ["openai"], "fallback": false }
}

With fallback on, a degraded primary silently reroutes to a secondary with different silicon, and your p99 becomes a blend of two distributions. That’s good for uptime, terrible for benchmark purity.

Architecture implications

Streaming is non-negotiable for interactive latency. Render the first token the moment it arrives; buffer nothing. For agentic loops, batch independent calls with asyncio rather than serial await to exploit provider concurrency.

Cache aggressively. Provider cache-control hints cut prefill cost and TTFT for repeated system prompts. Forward them:

client.chat.completions.create(
    model="gpt-5",
    messages=[{"role":"system","content":LONG_PROMPT}],
    extra_headers={"cache-control": "max-age=3600"}
)

If your gateway honors those hints, you turn a 4k-token prefill into a cache hit, which dominates TTFT more than raw model speed.

Decisive takeaway

Treat any GPT-5 speed benchmark as a distribution under your own load, not a vendor sticker. Measure p90/p99 TTFT and ITL with production-length prompts, disable fallback for the test, and multiply the resulting tokens/sec by your actual per-token price. If the interactive path needs sub-second feels, GPT-5 will likely deliver only if you cache prefill and stream; otherwise, a cheaper prior-generation model may win the cost-speed frontier. Ship the benchmark harness before you ship the model dependency.