n4nAI
Open console →
Benchmarks & performanceAnalysisGPU Inference Benchmarks

H200's extra memory bandwidth: does it cut latency?

Does the H200's extra memory bandwidth cut inference latency? We analyze H200 memory bandwidth inference latency versus H100 for LLM decode, prefill, and batch tradeoffs.

n4n Team4 min read977 words

Audio narration

Coming soon — every post will get a voice note here.

The H200 ships 4.8 TB/s of HBM3e bandwidth against the H100 SXM’s 3.35 TB/s, a 43% increase that sounds like a free latency win. For H200 memory bandwidth inference latency on large language models, the reality is nuanced: the extra bandwidth directly shrinks decode-phase token latency when the model is memory-bound, but it leaves prefill, compute-bound small models, and saturated throughput servers essentially unchanged. This analysis separates the cases where the upgrade pays off from where it is a line item without a corresponding speedup.

Why transformer decode is memory-bound

Autoregressive inference splits into two phases. Prefill processes the prompt in parallel; decode generates one token at a time. In decode, the batch dimension is typically small (often 1–32) while the weight matrices stay full-size. Each generated token requires reading the entire model weight set from HBM into the tensor cores, doing a relatively small amount of math, and writing back activations.

The arithmetic intensity—FLOPs per byte moved—is low. A 70B-parameter model in FP16 is 140 GB. If you generate a token with batch size 1, you move ~140 GB through the memory bus but perform only ~140G FLOPs of matmul (ignoring attention). At 1 petaFLOP/s of effective FP16 compute, that math takes ~0.14 ms; the data movement dominates.

Theoretical H200 memory bandwidth inference latency floor

We can put a lower bound on per-token latency by assuming perfect bandwidth utilization and zero compute overlap:

# Lower-bound decode latency from weight streaming alone
weights_gb = 140  # 70B params * 2 bytes
bandwidth_tb_s = {"H100_SXM": 3.35, "H200": 4.8, "H100_PCIe": 2.0}

for gpu, bw in bandwidth_tb_s.items():
    # bw * 1000 = GB/s
    ms_per_token = weights_gb / (bw * 1000) * 1000
    print(f"{gpu}: {ms_per_token:.1f} ms/token (weight-stream floor)")

This prints roughly:

H100_SXM: 41.8 ms/token
H200: 29.2 ms/token
H100_PCIe: 70.0 ms/token

That 29 ms is not a benchmark—it is the physics of H200 memory bandwidth inference latency when the model is purely bandwidth-limited. Real kernels add attention, layernorm, sampling, and launch overhead, so observed numbers are higher. But the ratio holds: a 43% bandwidth bump yields a ~30% reduction in the memory-bound floor.

Prefill does not care about bandwidth

Prefill batches all prompt tokens into large matrix multiplies. Arithmetic intensity scales with sequence length. For a 512-token prompt on the same 70B model, you reuse weights across 512 positions, so each weight byte serves ~512 FLOPs instead of 1. That pushes the bottleneck onto the tensor cores, not the memory bus.

H200 and H100 share the same Hopper compute profile: identical FP16/BF16/FP8 throughput. For prefill-heavy workloads (long documents, RAG ingestion, batched embedding), the extra 1.45 TB/s is wasted. Time-to-first-token (TTFT) is governed by compute and kernel fusion, not by HBM bandwidth.

Batch size moves you off the bandwidth cliff

The decode phase only stays memory-bound at small batch. Increase the batch dimension and weight reads are amortized: you load the weights once per step but multiply by B times more activations. Arithmetic intensity rises linearly with batch.

At batch 64, the same 140 GB weight load now supports 64× the matmul work. The compute units become the limit, and H200’s identical FLOPS mean per-token latency curves converge with H100. In a throughput-oriented serving setup (high concurrent streams, large continuous batching), you are buying memory bandwidth you cannot consume per request—you are already compute-saturated.

Worse, if you use the H200’s larger 141 GB capacity to hold a bigger KV cache and push batch size even higher, you may increase latency per token because the scheduler queues more sequences behind the same compute.

The capacity win is real even if latency isn’t

H200 carries 141 GB of HBM3e versus 80 GB on H100. That is not bandwidth, but it changes deployment shape. A 70B FP16 model alone leaves only ~10 GB on H100 for KV cache—useless for long contexts. On H200, you keep the model resident with ~70 GB headroom, enabling 32K+ context without layer offload or paging.

For inference gateways, this means fewer weird OOM-induced fallbacks to smaller models or CPU. When we route requests at n4n.ai across provider fleets, an H200-backed worker can honor a client’s routing directive to keep a 70B-class model hot with large context, while the per-token usage metering stays aligned because token count—not memory size—drives cost.

Software must exploit the bus

Extra bandwidth is only realized if the kernel issues coalesced reads and hides latency. Vanilla PyTorch eager mode with naive layer loops leaves 30–40% of peak bandwidth on the floor. You need:

  • Fused MLP and QKV kernels (e.g., CUTLASS or vendor fused modules)
  • FlashAttention-2/3 for the attention pass (memory-bound but bandwidth-sensitive)
  • CUDA graphs to remove launch overhead that masks the gain
  • FP8 weights if supported, which halve weight bytes and double effective bandwidth per param

Without these, H200 looks like an expensive H100 because the bottleneck becomes kernel launch or compute, not HBM.

Where the H200 earns its keep

Buy H200 for interactive, low-batch, large-model serving:

  • Single-user coding assistants (batch 1–4, 70B–120B)
  • Latency-sensitive chat where inter-token latency (TPOT) matters more than TTFT
  • On-device-style cloud inference where you cannot batch strangers due to privacy

In these regimes, H200 memory bandwidth inference latency drops from ~42 ms/token to ~29 ms/token theoretical, and real-world TPOT often follows the same ~25–30% trend.

Skip H200 when:

  • You run 7B–13B models (weights fit in cache, compute-bound)
  • You serve at batch ≥ 32 with continuous batching (compute-bound)
  • Prefill dominates (document summarization, bulk classification)

Measuring it in your own stack

Do not trust spec sheets. Benchmark the exact model and batch:

# Example using a workload generator (pseudo, not a real CLI)
# Measure TPOT at batch 1, seq 256, gen 128
gpu_bench --model meta-llama/70b --batch 1 --gpu H200 --metric tpot
gpu_bench --model meta-llama/70b --batch 1 --gpu H100 --metric tpot

Compare the median TPOT, not the mean. Then repeat at your production batch size. If the delta collapses at your batch, the bandwidth upgrade is not your lever.

Takeaway

The H200’s extra memory bandwidth cuts inference latency only when the decode phase is memory-bound: small batch, large weights, efficient kernels. Under those conditions, H200 memory bandwidth inference latency improves roughly 30% over H100 SXM, a real and measurable win for interactive LLM serving. For prefill, small models, or throughput-saturated batches, it is the same silicon with more memory and a wider bus you cannot stall. Provision H200 for latency-sensitive large-model endpoints; keep H100 for compute-bound throughput, and let capacity—not bandwidth—drive the memory-size decision.

Tagsh200memory-bandwidthgpu-benchmarkinference-latency

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceGPU memory bandwidth and its effect on inference latencyAnalyzes how GPU memory bandwidth drives inference latency for LLMs, with roofline math, batch tradeoffs, and quantization strategies for engineers.Benchmarks & performanceH100 vs H200 vs B200: inference latency comparedPractical comparison of H100 vs H200 vs B200 inference latency across memory, cost, throughput, and ops, with a verdict for production LLM serving.Benchmarks & performanceB200 NVLink and its impact on large model inferenceAnalyzing how B200 NVLink bandwidth reshapes large model inference: intra-node tensor parallelism, latency tradeoffs, and when the hardware actually pays off.Benchmarks & performanceB200 inference benchmarks for Llama 3.3 70BA practical analysis of B200 Llama 3.3 70B inference benchmark results: real throughput, latency tradeoffs, and when Blackwell beats Hopper.Benchmarks & performanceChoosing a GPU for LLM inference: H100 vs H200 vs B200Practical guide to selecting the best GPU for LLM inference H100 H200 B200: compare VRAM, bandwidth, and real-world tradeoffs for production.Benchmarks & performanceH100 SXM vs PCIe: inference latency differencesPractical comparison of H100 SXM vs PCIe inference latency differences: raw compute, NVLink, thermals, and cost tradeoffs for LLM serving.

All gpu inference benchmarks posts →