Latency vs throughput: what's the difference for LLMs?

When engineers talk about latency vs throughput llm performance, they’re often talking past each other. Latency measures how long a single request takes from send to finish; throughput measures how many requests your system completes per unit time. They pull in opposite directions: batching improves throughput but hurts latency, while optimizing for latency often leaves compute idle. Understanding this tension is the prerequisite for every capacity planning and routing decision you’ll make.

What latency actually measures

Latency in LLM inference breaks down into three distinct phases. Time to first token (TTFT) is the wall-clock delay between sending a request and receiving the first generated token — this dominates perceived responsiveness for streaming chat. Inter-token latency (ITL) is the gap between successive tokens once generation starts; it determines whether the stream feels smooth or stutters. End-to-end latency is the sum: TTFT plus (tokens generated × ITL), plus any queue time before the model even sees your request.

# Latency components you can measure client-side
import time

start = time.time()
first_token_time = None
tokens = []

async for chunk in stream_response(request):
    if first_token_time is None:
        first_token_time = time.time()
        ttft = first_token_time - start
    tokens.append(chunk)

e2e_latency = time.time() - start
itl = (e2e_latency - ttft) / len(tokens) if tokens else 0

Queue time is the silent killer. At high utilization, requests sit in a scheduler queue before the GPU gets to them. A model that generates at 50 ms/token can still deliver 5-second TTFT if the queue backs up. This is why latency percentiles (p50, p95, p99) matter more than averages — your users experience the tail.

What throughput actually measures

Throughput is tokens per second (or requests per second) sustained across the system. It’s a capacity metric, not a speed metric. A single H100 running Llama-3-70B might achieve 2,000 tokens/sec aggregate throughput — but if you feed it one request at a time, that same GPU might only deliver 200 tokens/sec because the model sits idle waiting for memory bandwidth between kernel launches.

The key distinction: throughput scales with concurrency up to the hardware’s saturation point. Latency degrades with concurrency once you exceed that point. The knee of the curve is where you want to operate.

# Throughput measurement at the server level
from prometheus_client import Counter, Histogram

TOKENS_GENERATED = Counter("llm_tokens_generated_total", "Total tokens")
REQUEST_LATENCY = Histogram("llm_request_latency_seconds", "E2E latency")

def record_request(token_count: float, latency: float):
    TOKENS_GENERATED.inc(token_count)
    REQUEST_LATENCY.observe(latency)

# Aggregate throughput = sum(tokens) / window_seconds

The fundamental conflict

Batching is the lever that moves both metrics in opposite directions. Continuous batching (also called iteration-level scheduling) lets the scheduler evict finished sequences and admit new ones each forward pass. This maximizes GPU utilization — hence throughput — but adds queue delay for incoming requests, inflating TTFT.

Static batching pads all sequences to the same length and processes them in lockstep. Predictable latency, terrible throughput when sequence lengths vary (which they always do).

# Conceptual continuous batching loop
while active_sequences:
    # 1. Append new requests up to max_batch_tokens
    # 2. Run single forward pass
    # 3. Sample next tokens for all active sequences
    # 4. Evict finished (EOS or max_len), admit waiting
    # 5. Repeat
    pass

The conflict shows up in KV cache pressure too. Longer context = more KV cache per sequence = fewer concurrent sequences fit in VRAM = lower max throughput. But truncating context to fit more sequences hurts quality. There’s no free lunch.

Comparison across dimensions

Where the metrics diverge in practice

Streaming chat

Users perceive TTFT as “intelligence speed.” A 2-second TTFT feels broken even if ITL is 20 ms. For this workload, you cap concurrency per GPU, accept 30–50% utilization, and over-provision. Speculative decoding with a small draft model (e.g., 7B drafting for 70B) can cut TTFT 2–3× with minimal quality loss.

Batch embedding / classification

No human waits. You want maximum tokens/sec/dollar. Pack sequences to the context limit, use static or large continuous batches, run INT8 or INT4 quantization, and drive GPUs to 90%+ utilization. Latency p99 can be 30 seconds — nobody cares.

RAG pipelines with tight SLAs

You have a chain: embed → retrieve → rerank → generate. The generate step inherits latency budget from upstream. If retrieval takes 400 ms and your SLA is 1 second, generation gets 600 ms max. This forces latency-optimized config even if throughput suffers.

Multi-tenant serving

You’re running both workloads on the same cluster. The answer is priority-aware scheduling: latency-sensitive requests jump the queue, throughput work fills gaps. n4n.ai’s routing layer exposes this via priority headers so you don’t need separate deployments.

Quantization and kernel choices

Quantization shifts the latency/throughput curve. INT4 GPTQ reduces VRAM per token ~4× vs FP16, letting you run 4× batch size — huge throughput win. But dequantization overhead adds ~1–2 ms/token latency. For latency-critical paths, BF16 or FP8 (on H100) often beats INT4 because the kernel stays memory-bound, not compute-bound.

FlashAttention-2 and PagedAttention change the slope. They reduce KV cache fragmentation and let you pack more sequences without OOM. The throughput gain is real; the latency impact is neutral to slightly positive because prefill kernels are more efficient.

Autoscaling signals that don’t lie

Don’t scale on GPU utilization alone. A GPU at 95% utilization with 10-second queue depth is failing latency SLAs. Scale on:

# Latency-driven scale-up
histogram_quantile(0.99, rate(llm_request_latency_seconds_bucket[2m])) > 2.0

# Throughput-driven scale-up (batch workloads)
rate(llm_tokens_generated_total[5m]) / count(gpu_available) > 1500

# Combined: queue depth per GPU
llm_queue_depth / count(gpu_ready) > 8

The combined signal catches both regimes. Latency workloads trigger on tail latency; throughput workloads trigger on token rate per GPU.

Which to choose

Choose latency optimization when:

• Humans wait for the response (chat, coding, search)
• You have a hard SLA on TTFT or p99 e2e latency
• Request patterns are bursty and unpredictable
• Quality at long context matters more than cost

Choose throughput optimization when:

• Work is offline, batched, or async (evals, embeddings, data labeling)
• Cost per million tokens is the primary KPI
• Request volume is steady and predictable
• You can tolerate minutes of queue time

Choose hybrid (most production systems):

• Route by priority: priority=low → throughput pool, priority=high → latency pool
• Use continuous batching with a max-queue-depth guard that sheds or reroutes excess
• Expose the trade-off to callers via a latency_budget_ms hint; the gateway picks the pool

The latency vs throughput llm decision isn’t a one-time architecture choice — it’s a runtime policy you enforce per request. Build the knobs, measure both metrics religiously, and let your routing layer make the call.