Benchmarking LLMs: throughput vs latency vs cost tradeoffs

The dominant failure mode in LLM benchmarking is measuring throughput, latency, and cost in isolation. The throughput vs latency vs cost llm tradeoff is a single coupled surface, and ignoring that coupling produces numbers that look authoritative but predict nothing about production behavior.

The myth of the single-number benchmark

A vendor sheet quoting “120 tokens/sec” tells you nothing about what happens when ten clients hit the same model concurrently. Latency at batch size 1 is a different physical regime from throughput at saturation. Prefill dominates time-to-first-token (TTFT); decode dominates tokens-per-second (TPS) under load.

Engineers often report p50 latency from a single-threaded loop. That hides the queueing delay that appears at concurrency > 1. Conversely, a throughput benchmark that fires requests as fast as possible with massive batching reports a cost-per-1k-tokens that no interactive user will ever see, because the p99 latency is seconds.

The throughput vs latency vs cost llm relationship is governed by GPU memory bandwidth, scheduler fairness, and pricing tiers that penalize long context. You cannot optimize one axis without moving the others.

A 70B parameter model at FP16 needs roughly 140GB of VRAM, forcing tensor parallelism across multiple GPUs. Decode throughput per request drops as the kv-cache grows and memory bandwidth saturates. A benchmark that does not state the concurrency and context length is not a benchmark; it is a press release.

Workload shape dictates the operating point

Before writing a benchmark, classify your traffic.

Interactive assistants

Users expect a response to start within 300–500 ms and stream at a readable pace. Here, TTFT p95 matters more than aggregate TPS. A model that delivers 40 TPS but has 2 s prefill is worse than one at 20 TPS with 200 ms prefill. Perceived latency is TTFT plus the spacing of first few chunks, not the steady-state generation rate.

Batch pipelines

Nightly document extraction couldn’t care less about latency. It wants maximum tokens processed per dollar. You should drive concurrency until the provider rate-limits you, then measure effective throughput and error rate. The relevant metric is cost per million output tokens at sustained load, not the latency of a single call.

Bursty API traffic

Real APIs see Poisson arrivals with occasional spikes. Benchmarks using constant-rate injection overestimate capacity. You need headroom; the knee of the latency curve under burst determines how much you over-provision. If your p95 TTFT doubles when concurrency goes from 10 to 20, you are already past the knee.

Measuring the joint curve

You need a load generator that ramps concurrency and records per-request TTFT, total tokens, and status. Use the OpenAI client against any compatible endpoint. Below is a minimal async Python harness.

import asyncio, time, openai

async def single(client, model, prompt, max_tokens):
    t0 = time.perf_counter()
    resp = await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
    )
    dt = time.perf_counter() - t0
    return dt, resp.usage.total_tokens

async def ramp(model, prompt, concurrency, requests):
    client = openai.AsyncOpenAI()  # set base_url=... for a gateway
    sem = asyncio.Semaphore(concurrency)
    async def wrapped(p):
        async with sem:
            return await single(client, model, p, 128)
    tasks = [wrapped(prompt) for _ in range(requests)]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    latencies = [r[0] for r in results if not isinstance(r, Exception)]
    tokens = sum(r[1] for r in results if not isinstance(r, Exception))
    latencies.sort()
    p95 = latencies[int(0.95 * len(latencies))]
    return p95, tokens

# asyncio.run(ramp("gpt-4o-mini", "Summarize: " + "x"*200, 8, 100))

Run this at concurrency 1, 4, 8, 16, 32. Plot p95 latency on the y-axis, throughput (tokens/sec = total_tokens / wall_clock) on the x-axis, and annotate each point with cost = tokens * price_per_token. That graph is the throughput vs latency vs cost llm surface for your workload.

When comparing providers, a gateway that provides per-token usage metering and automatic fallback (such as n4n.ai) keeps cost accounting honest and prevents a flaky provider from skewing latency numbers during the ramp.

Define your metrics precisely:

• TTFT: wall-clock from request send to first byte of response.
• TPS: output_tokens / (total_latency - TTFT).
• Total latency: TTFT + output_tokens / TPS.
• Effective cost: (input_tokens + output_tokens) * price, plus retry overhead.

Cost is not just price per token

Published token prices ignore retries. If a model fails 5% of requests at your concurrency, your effective cost rises by at least that margin, plus the latency of the retry. Most providers price output tokens 2–4x higher than input tokens, so your output length distribution drives cost more than input size.

Some providers charge separately for cached context; if your benchmark re-sends the same system prompt, you must forward cache-control hints or you will overstate cost. Context length silently kills throughput. A 32k input requires prefill compute linear in tokens; TTFT grows even if decode stays constant. Measure with your real prompt distribution, not a 32-token toy.

Levers that move the surface

• Model size: Smaller models (e.g., 8B) decode faster and cost less per token but may need more attempts to meet quality bars.
• Quantization: INT4/INT8 reduces memory bandwidth pressure, raising TPS at slight quality cost.
• Streaming: Lowers perceived latency (first chunk arrives early) but does not change generation cost.
• Batching: Provider-side batching improves throughput but worsens tail latency for isolated requests.
• Routing: Directing requests to the cheapest healthy provider based on real-time degradation shifts the cost axis without touching latency.

The throughput vs latency vs cost llm decision is therefore a configuration problem, not a model-picking problem alone. You tune concurrency, batch size, and routing alongside model choice.

A decisive benchmarking protocol

1. Define the SLO. Example: p95 TTFT < 800 ms, p95 total latency < 3 s, cost < $0.50 per 1k requests of 1k in / 200 out tokens.
2. Capture real payloads. Sample production prompts; preserve length distribution and mix of system/user turns.
3. Ramp concurrency. Find the point where p95 latency breaches SLO. That is max sustainable throughput for that model/config.
4. Compute effective cost at that point including retries, cache hits, and price tiers.
5. Compare candidates on the same harness. The winner is the one that meets SLO at lowest cost, not the one with best isolated latency or best saturated throughput.

Skip any public benchmark that does not report the concurrency level, input/output lengths, and price assumptions. Those omissions mean the numbers are not reproducible for your case.

Takeaway

Treat throughput, latency, and cost as a single coupled surface measured under your own workload shape. Run a concurrency ramp, record p95 TTFT and tokens/sec, and price the result with real metering. The model that wins is the one that sits at the knee of your curve—meeting the latency SLO while minimizing dollars per useful token. Anything else is a vanity metric.