How memory bandwidth affects LLM token generation speed

Memory bandwidth LLM inference speed is the single most overlooked constraint when engineers try to push token throughput higher. You can buy the fastest H100s available, but if your workload is memory-bound — which almost all autoregressive decoding is — the compute units sit idle waiting for weights and KV cache to arrive. This guide walks through why bandwidth dominates decode, how to prove it on your own hardware, and the concrete levers you can pull to get more tokens per second per dollar.

The decode loop is a bandwidth problem

During prefill, the GPU computes a large matrix multiply: the entire prompt against the model weights. That’s compute-bound on modern GPUs. But decode is different. Each new token requires reading the full model weights (or the active expert subset) plus the growing KV cache from VRAM, doing a tiny matvec per layer, and writing the updated KV cache back. The arithmetic intensity — FLOPs per byte moved — drops by orders of magnitude.

For a 70B parameter model at FP16, weights alone are 140 GB. The KV cache for a 4k context window adds another ~10 GB. Generating one token means streaming ~150 GB through the memory subsystem. An H100 SXM5 delivers 3.35 TB/s peak bandwidth. At 100% utilization that’s ~22 tokens/s theoretical maximum. Real kernels achieve 60–75% of peak, so you see 13–16 tokens/s. The math is unforgiving: tokens per second ≈ (achieved bandwidth) / (bytes per token).

# Rough bytes-per-token estimate for a decoder-only transformer
def bytes_per_token(num_layers, hidden_size, num_kv_heads, head_dim, dtype_bytes=2, context_len=4096):
    # Model weights read once per token (all layers)
    weight_bytes = num_layers * (4 * hidden_size * hidden_size) * dtype_bytes  # QKV + O + MLP
    # KV cache read + write per token (2 * layers * 2 * kv_heads * head_dim * context)
    kv_bytes = num_layers * 2 * 2 * num_kv_heads * head_dim * context_len * dtype_bytes
    return weight_bytes + kv_bytes

# Llama-3-70B: 80 layers, 8192 hidden, 8 kv heads, 128 head_dim
print(f"Bytes/token: {bytes_per_token(80, 8192, 8, 128) / 1e9:.1f} GB")
# Output: ~150 GB/token

Prove you’re memory-bound before optimizing

Don’t guess. Profile with NCU (NVIDIA Nsight Compute) or nsys and look at two metrics: DRAM Throughput and SM Throughput. If DRAM Throughput > 80% and SM Throughput < 30%, you’re bandwidth-bound. If both are low, you have a kernel launch or synchronization problem. If SM is high and DRAM is low, you’re compute-bound (rare for decode).

# Quick sanity check: measure achieved memory bandwidth during decode
nsys profile --stats=true --force-overwrite true \
  -o decode_profile python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B-Instruct --max-model-len 4096 \
  --enforce-eager  # disable CUDA graphs to see raw kernel behavior

In the NCU log, check Memory Throughput under the Memory Workload Analysis section. Compare against the GPU’s spec sheet. On an H100, if you see 2.0 TB/s achieved vs 3.35 TB/s peak, you’re at 60% — typical for current kernels. On an A100 80GB (1.55 TB/s peak), 900 GB/s achieved is similarly normal.

Pitfall: Running with --enforce-eager disables CUDA graphs, which adds ~15–20% overhead from kernel launches. Use it for profiling, but production should use graphs. The bandwidth ceiling doesn’t change; graphs just reduce the fixed overhead per step.

Quantization is your first lever

Reducing dtype_bytes in the formula above linearly increases token throughput — if you stay memory-bound. INT4 (4-bit) cuts weight bytes by 4× vs FP16. INT8 cuts by 2×. The KV cache can also be quantized (KV8, KV4), though quality degrades faster there.

# vLLM quantization config examples
from vllm import EngineArgs

# AWQ INT4 weights, FP16 activations (recommended starting point)
engine_args = EngineArgs(
    model="meta-llama/Llama-3-70B-Instruct",
    quantization="awq",
    dtype="half",  # activation dtype
    max_model_len=4096,
)

# GPTQ INT4 alternative
engine_args = EngineArgs(
    model="meta-llama/Llama-3-70B-Instruct",
    quantization="gptq",
    dtype="half",
)

# FP8 (H100 only) - 2x weight reduction, native tensor cores
engine_args = EngineArgs(
    model="meta-llama/Llama-3-70B-Instruct",
    quantization="fp8",
    dtype="float8_e4m3fn",
)

Tradeoff: INT4 weights + FP16 activations typically recover 95–98% of FP16 quality on benchmarks. INT4 weights + INT8 KV cache pushes to 90–95%. FP8 on H100 is nearly lossless but requires Hopper. Always evaluate on your task — coding and reasoning degrade faster than chat.

KV cache management changes the bandwidth math

The KV cache grows with context length. At 4k context, KV cache is ~10 GB for 70B. At 32k, it’s ~80 GB — larger than the weights. This shifts the bottleneck: for long contexts, KV cache bandwidth dominates weight bandwidth.

Three techniques reduce KV cache pressure:

1. Sliding window attention — Only attend to the last W tokens. Reduces KV cache from O(context) to O(W). Mistral and Gemma use this natively.

# vLLM sliding window (model must support it)
engine_args = EngineArgs(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    max_model_len=32768,  # full context
    # sliding_window=4096  # set in model config, not engine
)

2. KV cache quantization — Store K/V in INT8 or INT4. vLLM supports kv_cache_dtype="fp8" on H100 (2× reduction) and experimental INT4.

engine_args = EngineArgs(
    model="meta-llama/Llama-3-70B-Instruct",
    quantization="awq",
    kv_cache_dtype="fp8",  # H100 only
)

3. Paged attention with block eviction — vLLM’s default paged attention already avoids fragmentation. For multi-turn conversations, evict old blocks when context exceeds budget.

# Custom block manager for aggressive eviction (vLLM 0.6+)
from vllm.sequence import SequenceGroupMetadata

class EvictingBlockManager:
    def __init__(self, max_blocks: int):
        self.max_blocks = max_blocks
    
    def allocate(self, seq_group: SequenceGroupMetadata) -> List[int]:
        if self.free_blocks < seq_group.num_required_blocks:
            self._evict_oldest(seq_group.num_required_blocks - self.free_blocks)
        return super().allocate(seq_group)

Pitfall: KV quantization + weight quantization compounds quality loss. Test FP8 KV + INT4 weights before deploying. On A100/Ampere, FP8 KV isn’t supported; INT8 KV is the floor.

Batching: the only way to amortize weight reads

Weight bytes are read once per token per request. If you serve 1 request, you stream 140 GB/token. If you serve 32 requests concurrently with continuous batching, you still stream 140 GB/token total — but you produce 32 tokens in that same memory transaction. Throughput scales linearly with batch size until you hit the compute ceiling or VRAM limit.

# vLLM continuous batching config for max throughput
engine_args = EngineArgs(
    model="meta-llama/Llama-3-70B-Instruct",
    quantization="awq",
    max_model_len=4096,
    max_num_seqs=256,           # max concurrent sequences
    max_num_batched_tokens=8192, # token budget per iteration
    # These two control the batch size vs latency tradeoff
)

The max_num_batched_tokens parameter is your primary throughput knob. Set it to fill VRAM: (VRAM - model_size) / (kv_bytes_per_token * context_len). For 70B AWQ (≈35 GB) on 8×H100 (640 GB), you have ~600 GB for KV. At 4k context, that’s ~75k tokens → batch ~75k/4k ≈ 18 sequences at full context, or hundreds at short context.

Pitfall: Large batches increase tail latency. If you need p50 < 100ms, cap max_num_batched_tokens lower and accept lower throughput. There’s no free lunch.

Tensor parallelism splits bandwidth, not compute

Tensor parallel (TP) shards weights across GPUs. Each GPU reads 1/TP_size of the weights per token. But it also requires an all-reduce after each layer’s attention and MLP. The all-reduce travels over NVLink (900 GB/s on H100, 600 GB/s on A100), not DRAM.

# 70B on 4×H100 with TP=4
# Each GPU holds 35 GB weights (INT4) + KV cache
# All-reduce per layer: 2 * hidden_size * TP_size * dtype_bytes
# = 2 * 8192 * 4 * 2 = 128 KB per layer per token
# 80 layers → 10 MB all-reduce per token
# At 900 GB/s NVLink: 0.011 ms per token overhead

Rule of thumb: TP reduces per-GPU memory bandwidth pressure linearly, but adds NVLink traffic. For decode, TP=4 on H100 is usually optimal for 70B. TP=8 helps only if single-GPU VRAM is insufficient. Never use TP > 1 on a single GPU — it adds overhead with no bandwidth benefit.

Pitfall: Pipeline parallelism (PP) is bad for decode latency. It adds bubble overhead and doesn’t reduce per-GPU bandwidth. Use PP only for prefill-heavy workloads or models too large for TP alone.

Hardware selection: bandwidth per dollar

Key insight: Bandwidth per dollar favors older/lower-tier GPUs if your model fits in VRAM. A 7B model on 4×A10G (96 GB total, 2.4 TB/s aggregate) costs less than 1×H100 and delivers more tokens/s. But you need tensor parallelism across 4 GPUs, adding NVLink traffic and complexity.

For 70B+ models, H100’s 3.35 TB/s and FP8 support are genuinely differentiating. The 2× bandwidth over A100 translates directly to 2× token throughput at same quantization.

Putting it together: a tuning checklist

When you deploy a new model or hardware config, run this sequence:

1. Baseline: Run single-request decode with enforce_eager, measure tokens/s and NCU DRAM throughput.
2. Enable CUDA graphs: Should gain 15–20% with same bandwidth.
3. Apply quantization: AWQ INT4 → measure quality, then throughput. Expect 3–3.5× speedup vs FP16.
4. Tune batch size: Increase max_num_batched_tokens until DRAM throughput saturates or latency SLO breaks.
5. Add TP: If single GPU OOMs or bandwidth saturated, add GPUs with TP. Re-measure.
6. KV cache optimization: If context > 8k, enable FP8 KV (H100) or sliding window (supported models).
7. Profile again: Verify DRAM throughput > 75% peak. If not, check kernel occupancy and launch config.

# Quick throughput benchmark script
import time
from vllm import LLM, SamplingParams

def benchmark_throughput(model_path, quantization, batch_sizes, context_len=4096, output_len=128):
    llm = LLM(model=model_path, quantization=quantization, max_model_len=context_len, enforce_eager=False)
    prompts = ["Hello " * (context_len // 10)] * max(batch_sizes)
    sampling = SamplingParams(max_tokens=output_len, temperature=0)
    
    for bs in batch_sizes:
        start = time.perf_counter()
        outputs = llm.generate(prompts[:bs], sampling)
        elapsed = time.perf_counter() - start
        total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
        print(f"Batch {bs}: {total_tokens/elapsed:.1f} tok/s, latency {elapsed/bs*1000:.0f} ms/req")

benchmark_throughput("meta-llama/Llama-3-70B-Instruct", "awq", [1, 4, 16, 64, 256])

Common pitfalls that waste bandwidth

1. Running FP16 on H100 without FP8. You paid for tensor cores that natively accumulate FP8. Leaving them idle wastes 2× potential throughput.

2. Setting max_model_len far above actual usage. KV cache allocates for the max. If you set 32k but average 2k, you’re reserving 16× VRAM for nothing — limiting batch size and throughput.

3. Ignoring CPU-GPU transfer overhead in disaggregated setups. If you offload KV cache to CPU (for extreme context), PCIe 5.0 x16 (128 GB/s) becomes the new bottleneck. Only viable for very low throughput, high context.

4. Using torch.compile or custom kernels without measuring. The default FlashAttention-2 / PagedAttention kernels in vLLM/TGI are already highly optimized. Custom kernels rarely beat them on bandwidth utilization unless you’re fusing multiple operations.

5. Assuming more GPUs = linear scaling. TP adds all-reduce. PP adds pipeline bubbles. Data parallel (multiple replicas) scales throughput but multiplies VRAM cost. Choose based on whether you’re latency-bound or throughput-bound.

The bandwidth ceiling is real — plan for it

You cannot optimize past the memory wall. Every token generated requires moving bytes from VRAM to compute. The only variables you control are: bytes per token (quantization, KV compression, model architecture), achieved bandwidth fraction (kernel quality, batch size), and hardware peak bandwidth (GPU selection).

For a given model and hardware, maximum tokens/s = (peak bandwidth × utilization) / bytes_per_token. All engineering effort goes into maximizing the numerator and minimizing the denominator. Quantization and batching are the two biggest levers; everything else is incremental.

When you hit the ceiling, the only remaining moves are: smaller model (distillation, pruning), different architecture (Mamba, RWKV, sliding window), or faster memory (HBM3e, next-gen GPUs). Know the ceiling before you promise latency SLAs.