GPU memory bandwidth inference latency is the pair of variables that dictates real-world token generation speed for transformer models, and it matters more than peak FLOPS for most serving workloads. The thesis of this analysis is simple: for batch size one, you are almost always waiting on weight movement, not matrix math, and architecting around that constraint wins more than buying faster calculators.
The roofline: why inference is memory-bound
Transformer decoding is a GEMV (general matrix-vector multiply) per layer. For a model with P parameters in FP16, each token requires reading P weights from HBM. The arithmetic intensity is roughly 2P FLOPs per 2P bytes = 1 FLOP/byte. That is far below the arithmetic intensity needed to saturate modern GPUs (often >10–50 FLOP/byte). You hit the memory roof, not the compute roof.
Calculate the lower bound:
param_bytes = 70e9 * 2 # 70B params, FP16 = 2 bytes
bandwidth_tb_s = 2.0 # A100 HBM2e effective
latency_s = param_bytes / (bandwidth_tb_s * 1e12)
print(latency_s * 1000) # ~70 ms per tokenThat 70 ms/token ceiling (about 14 tokens/s) matches empirical llama-70B-on-A100 numbers when not batching. H100’s 3.35 TB/s HBM3 pushes the same math to ~42 ms/token. GPU memory bandwidth inference latency scales nearly linearly here.
Prefill vs decode: two regimes
Prompt processing (prefill) multiplies a large input matrix by weights. Arithmetic intensity is high; you are compute-bound and saturated FLOPS matter. Autoregressive decode generates one token at a time. Each step loads the entire weight set for a single vector. Intensity collapses to ~1 FLOP/byte.
This split should drive your serving design. Prefill can be batched aggressively across many requests. Decode should stay on the fastest memory bus available, with minimal cross-device traffic.
Batch size and the crossover to compute-bound
At batch=1, weight fetch dominates. Increase concurrent requests B: the weights are still fetched once per iteration, then multiplied by a wider matrix. Iteration time stays near the memory-bound constant until compute catches up.
def iter_time(param_bytes, bw, batch, peak_flops, flops_per_param=2):
compute_flops = (param_bytes / 2) * flops_per_param * batch
mem_time = param_bytes / bw
compute_time = compute_flops / peak_flops
return max(mem_time, compute_time)
# A100: bw=2e12, peak_flops=300e12
print(iter_time(140e9, 2e12, 1, 300e12) * 1000) # ~70 ms
print(iter_time(140e9, 2e12, 64, 300e12) * 1000) # still ~70 msThroughput (total tokens/s) scales with B, but per-request latency does not improve—it may degrade via queuing. For chat completions with p99 < 200 ms, keep batch small and accept lower utilization.
The KV cache multiplier
Decode also reads and writes the KV cache. For a 70B model, 80 layers, hidden 8192, FP16:
kv_per_token = 2 * 80 * 8192 * 2 # ~2.6 MB
context = 32768
kv_total = kv_per_token * context # ~82 GBAt 32k context, KV traffic rivals weight traffic. Every generated token must fetch its prior KV slices. GPU memory bandwidth inference latency now includes this second stream, pushing you further from compute-bound even at modest batch. This is why long-context requests are slower per token despite identical compute.
Tensor parallelism and the bandwidth tax
When a model doesn’t fit on one GPU, you slice layers across N devices. Each forward step needs all-reduce of activations. On NVLink (600 GB/s aggregate class) this is tolerable; on PCIe 4.0 x16 (64 GB/s) it is a tax that can eclipse weight-fetch savings.
A 70B model sharded across two A100s over PCIe still moves activations of size batch * seq * 8192 * 2 each direction. At seq=2048 that is ~67 MB per layer—small vs 140 GB weights, but synchronization stalls the pipeline. For single-stream low-latency, prefer one large GPU with quantization over two smaller ones.
Quantization: the cheapest bandwidth multiplier
FP8 or INT8 cuts param_bytes by 2× or 4×. On H100, FP8 is native; effective bandwidth for weight movement doubles versus FP16 because each byte carries 2× info. The same 70B model in FP8 is 70 GB, fitting on one H100, with theoretical 21 ms/token.
Accuracy cost is real: calibration needed, some layers sensitive. But for many instruction models, INT4 via GPTQ loses <1% on common benchmarks. The latency win is pure GPU memory bandwidth inference latency reduction.
# Illustrative quantization step (use your own pipeline)
python -m torch.quantization.quantize_fx --model llama-70b --dtype fp8 --output model_fp8Measuring it in your own stack
Don’t trust specs; measure. A minimal PyTorch probe:
import torch
a = torch.randn(8192, 8192, device='cuda', dtype=torch.float16)
b = torch.randn(8192, device='cuda', dtype=torch.float16)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for _ in range(100):
c = a @ b
end.record()
torch.cuda.synchronize()
print(start.elapsed_time(end) / 100) # ms per opIf measured ms/token is close to param_bytes / bandwidth, you are bandwidth bound. If far above, fix framework overhead (CUDA graphs, kernel launch, Python loops) first.
Gateway-level routing and degradation
At scale, you rarely control the physical GPU per request. An inference gateway can route to instances backed by high-bandwidth silicon when the client passes a latency hint. n4n.ai honors client routing directives and forwards provider cache-control hints, so a request tagged for low latency can avoid a provider running narrower-bandwidth GPUs behind noisy neighbors. Automatic fallback also masks a provider whose HBM is saturated. This is the system-level acknowledgment that GPU memory bandwidth inference latency variance is a property of shared hardware.
Takeaway
Buy GPUs by bandwidth per watt and per dollar, not TFLOPS. Quantize aggressively to shrink the working set. Keep batch size minimal for interactive latencies; scale with separate batch pipelines for offline jobs. Shard only when a single device cannot hold the model in FP8. If you internalize that weight movement is the tax, every architectural choice gets simpler.