What is tokens per second in LLM benchmarking?

Tokens per second (TPS) is the rate at which a language model generates output tokens during inference, typically measured over the steady-state portion of generation after the first token arrives. It quantifies throughput rather than latency — how many tokens the model produces per unit of time once the pipeline is full. Engineers use TPS to compare model serving performance, size GPU fleets, and set SLAs for streaming applications.

How tokens per second is calculated

The basic formula is straightforward:

TPS = (total_output_tokens - 1) / (time_last_token - time_first_token)

Subtracting one token and using the interval between first and last token excludes the prefill phase, which has different compute characteristics than decode. Prefill processes the entire prompt in parallel; decode generates tokens sequentially, one at a time, with each step attending to all previous tokens.

In practice, you’ll see two variants reported:

Aggregate TPS — Total tokens generated across all concurrent requests divided by wall-clock time. This measures server throughput.

Per-request TPS — Tokens generated for a single request divided by its decode duration. This measures user-perceived speed.

import time
from dataclasses import dataclass

@dataclass
class GenerationMetrics:
    prompt_tokens: int
    completion_tokens: int
    time_to_first_token: float  # seconds
    total_generation_time: float  # seconds

def calculate_tps(metrics: GenerationMetrics) -> float:
    """Calculate per-request tokens per second (decode phase only)."""
    if metrics.completion_tokens <= 1:
        return 0.0
    decode_time = metrics.total_generation_time - metrics.time_to_first_token
    if decode_time <= 0:
        return float('inf')
    return (metrics.completion_tokens - 1) / decode_time

# Example: 500 completion tokens, 200ms TTFT, 2.3s total
m = GenerationMetrics(
    prompt_tokens=1200,
    completion_tokens=500,
    time_to_first_token=0.2,
    total_generation_time=2.3
)
print(f"TPS: {calculate_tps(m):.1f}")  # ~238 tokens/sec

Why TPS matters for production systems

Capacity planning

TPS directly determines how many concurrent users a GPU can serve. If a model runs at 100 TPS per request and your average completion is 800 tokens, each request occupies the GPU for 8 seconds of decode time. With 80 GB H100s running a 70B model at 4-bit quantization, you might achieve 2,000 aggregate TPS. That translates to roughly 20 concurrent streams at 100 TPS each — or fewer if requests have longer contexts that reduce batch efficiency.

Streaming UX thresholds

Human reading speed tops out around 20–30 tokens per second (roughly 300–500 words per minute). Generating faster than this wastes compute unless you’re feeding another model or writing to storage. Generating slower creates visible “typewriter lag” that degrades perceived quality, especially in chat and coding assistants.

Cost modeling

Provider pricing often correlates with TPS capability. Faster models (smaller, quantized, or running on newer hardware) cost less per million tokens because they amortize GPU-hours over more output. When evaluating providers, divide price per million tokens by their sustained TPS to get a rough $/TPS-hour figure for comparison.

Concrete example: comparing two deployments

Consider two ways to serve Llama-3.1-70B:

Deployment A has higher aggregate throughput — better for high-volume batch workloads like document processing. Deployment B has higher per-request TPS — better for interactive chat where users wait on single streams.

The difference comes from batching efficiency. Large batches saturate tensor cores but increase per-request latency due to scheduler overhead and memory pressure. Small batches keep latency low but underutilize compute. Most production systems run a mix: dedicated low-batch pools for interactive traffic, high-batch pools for async workloads.

# Example router config for traffic splitting
routing:
  interactive:
    model: "llama-3.1-70b-fp8"
    pool: "h100-low-batch"
    max_batch_size: 4
    target_tps_per_request: 120
  batch:
    model: "llama-3.1-70b-fp8"
    pool: "h100-high-batch"
    max_batch_size: 64
    target_aggregate_tps: 4000

Common misconceptions

TPS equals latency

TPS and latency are related but distinct. Time-to-first-token (TTFT) measures latency — how long until the user sees anything. TPS measures sustained throughput after that. A system can have excellent TTFT (50ms) but poor TPS (20 tokens/sec), making long responses feel slow. Conversely, high TPS with terrible TTFT feels unresponsive initially.

Both matter. For chat, optimize TTFT first (target <200ms), then ensure TPS exceeds reading speed. For autocomplete, TTFT is everything — TPS barely matters because completions are short.

Higher TPS always means better model

TPS is a property of the serving stack, not the model weights alone. The same model weights can yield 50 TPS or 500 TPS depending on:

• Quantization (FP16 → INT4 can 2–3× TPS with minimal quality loss)
• Kernel optimization (FlashAttention, PagedAttention, speculative decoding)
• Hardware (H100 vs A100 vs consumer GPU)
• Batch size and scheduling policy
• Context length (longer context → lower TPS due to KV cache pressure)

Benchmark marketing often quotes peak TPS at batch=1, short context, INT4 on H100. Your production numbers will be lower.

TPS is constant during generation

In autoregressive decoding, TPS typically decreases as sequence length grows. Each new token attends to all previous tokens, so the KV cache grows linearly and attention computation grows quadratically (though FlashAttention reduces this to linear memory, compute still scales with sequence length).

Position 1:  150 TPS
Position 500: 140 TPS
Position 2000: 110 TPS
Position 4000:  85 TPS

This decay is steeper for models without grouped-query attention and for longer contexts. When benchmarking, report TPS at multiple output lengths or specify the measurement window.

Aggregate TPS predicts per-request latency

Aggregate TPS = (concurrent_requests × per_request_TPS) only holds under perfect batching with no scheduling overhead. Real systems have:

• Padding waste (requests in a batch wait for the longest)
• Scheduler latency (queue time, kernel launch overhead)
• Memory fragmentation (KV cache allocation gaps)
• Preemption costs (priority inversion, context switching)

At high utilization, per-request TPS drops 15–30% below the theoretical aggregate / concurrency ratio. Load test at your target concurrency, not just batch=1.

Measuring TPS correctly

Warm-up runs

First requests after model load are slower due to kernel compilation (CUDA graphs, Triton JIT), KV cache allocation, and CPU-GPU synchronization. Discard the first 10–20 requests per worker.

Steady-state window

Measure over at least 100 completion tokens, ideally 500+. Short completions (<50 tokens) are dominated by TTFT variance and kernel launch overhead.

Control variables

Document these for any benchmark:

{
  "model": "llama-3.1-70b",
  "quantization": "fp8",
  "hardware": "4x H100 80GB NVLink",
  "batch_size": 8,
  "input_tokens": 1024,
  "output_tokens": 512,
  "concurrency": 8,
  "scheduler": "vllm 0.6.3, chunked_prefill=2048",
  "kv_cache_dtype": "fp8",
  "speculative_decoding": false
}

Percentiles over averages

Report p50, p90, p99 per-request TPS. Averages hide tail latency that causes user-visible stalls. If p99 TPS is half of p50, your scheduler or memory management needs work.

TPS in the context of other metrics

For streaming chat, the UX formula is roughly: perceived_speed = min(TPS, reading_speed) after TTFT. Optimize TTFT until it’s imperceptible (<100ms), then ensure TPS > 30. Beyond that, invest in quality, not speed.

Practical takeaways

1. Define your workload first. Batch document processing needs aggregate TPS. Interactive chat needs per-request TPS and low TTFT. They require different hardware configurations.

2. Quantization is the highest-leverage knob. Moving from FP16 to INT4 or FP8 typically doubles TPS with <1% quality degradation on modern models. Do this before buying more GPUs.

3. Speculative decoding can 2–3× TPS for compatible workloads (short contexts, greedy sampling). It adds complexity — draft model management, verification overhead — but pays off at scale.

4. Monitor TPS in production, not just benchmarks. Real traffic has variable context lengths, priority classes, and burst patterns that synthetic loads miss. Alert on p99 TPS degradation.

5. Route by TPS requirements. n4n.ai’s routing directives let you send latency-sensitive traffic to low-batch pools and throughput-insensitive traffic to high-batch pools, maximizing fleet utilization without sacrificing UX.

TPS is a serving metric, not a model metric. The same weights serve at wildly different speeds depending on your stack. Measure your stack, not the vendor’s marketing page.