Llama 3.1 405B inference: H100 vs H200 throughput

Engineering teams sizing inference for the 405B model quickly hit a hardware question: what does Llama 3.1 405B H100 vs H200 throughput look like once you account for tensor parallelism, KV cache, and batch size? The two GPUs share the same GH100 die and NVLink topology, but the H200’s HBM3e capacity and bandwidth change the math for a model this large.

Hardware fundamentals

Memory capacity and bandwidth

The Llama 3.1 405B weights consume ~810 GB in FP16 and ~405 GB in FP8. An H100 SXM ships with 80 GB HBM3 (3.35 TB/s); an H200 ships with 141 GB HBM3e (4.8 TB/s). Eight H100s provide 640 GB total—insufficient for FP16 weights, so you either quantize to FP8 or span 16 GPUs. Eight H200s provide 1128 GB, holding FP16 weights with headroom for KV cache and activations on a single node.

That single-node capability is the biggest differentiator. It removes cross-node latency from the critical path for weight fetching and simplifies orchestration.

Compute and networking

Both GPUs deliver identical FLOPS (around 990 TFLOPS FP16 dense, 1979 TFLOPS FP8). NVLink bandwidth per GPU is 900 GB/s on both. For prefill-heavy workloads, compute-bound phases see no difference. The gap appears in memory-bound decode and in how many requests you can pack before hitting capacity limits.

Throughput characteristics

Decode-bound vs prefill-bound

Autoregressive decode reads the full weight matrix per token. Theoretical max tokens/sec per GPU for weight-only reads:

def theoretical_decode_tps(bandwidth_tb_s, param_gb, bytes_per_param=2):
    bw = bandwidth_tb_s * 1e12
    size = param_gb * 1e9 * bytes_per_param
    return bw / size

# H100 FP8 (405GB)
print(theoretical_decode_tps(3.35, 405, 1))  # ~8.3 tok/s
# H200 FP16 (810GB)
print(theoretical_decode_tps(4.8, 810, 2))   # ~5.9 tok/s

Real numbers are lower due to attention, sampling, and kernel overhead, but the ratio holds: H100 with FP8 wins per-GPU raw decode, while H200 with FP16 closes the gap and avoids quantization.

KV cache pressure

Llama 3.1 405B uses grouped-query attention (8 KV heads, head_dim 128, 126 layers). KV cache per token:

layers=126; kv_heads=8; head_dim=128; bytes=2
per_token = 2 * layers * kv_heads * head_dim * bytes
print(per_token)  # 516096 bytes ~0.49 MB/token
seq=2048
print(per_token*seq / 1e6)  # ~1.02 GB per request at 2K context

Batch 64 needs ~65 GB of KV cache cluster-wide. With 8-way tensor parallel, each H200 carries ~101 GB weights (FP16) plus its KV share; each H100 FP8 carries ~50 GB weights plus KV share. H200’s total node capacity absorbs more concurrent sessions before spilling.

Real-world batching

At batch size 64+ and sequence length 2K, H200’s extra capacity lets you keep KV cache resident for more concurrent sessions. On H100 you spill to host memory or reduce batch, dropping effective throughput per dollar. For Llama 3.1 405B H100 vs H200 throughput under mixed load, the H200 typically sustains higher aggregate tokens/sec per node because it isn’t fighting capacity evictions.

Cost model

Capex and cloud rates

H200 boards cost more per unit (roughly 15–25% premium in cloud hourly pricing as observed on major providers). An 8x H100 node on typical cloud runs ~$20–30/hr; 8x H200 ~$28–40/hr depending on region. If H100 requires 16 GPUs to hold FP16, the H200 single-node wins on rack space and interconnect.

Total cost per million tokens

Because the H200 serves more tokens per node-hour at large batch, its effective cost per million output tokens can be lower despite higher hourly rate. For FP8-quantized deployments where H100 fits on 8 GPUs, the H100 remains cheaper per token at small batch.

Ergonomics and ops

Driver and framework support

Both expose the same CUDA 12.x surface. vLLM, TensorRT-LLM, and Hugging Face TGI support H200 out of the box. The only ops difference is setting --gpu-memory-utilization higher on H200 because you have 141 GB to play with.

vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 8192

On H100 with FP8 you must pass --quantization fp8 and verify calibration.

Multi-node scaling

If you need 16 H100s, you pay for InfiniBand and orchestration complexity. H200’s single-node footprint reduces this. A gateway such as n4n.ai abstracts backend differences behind one OpenAI-compatible endpoint and automatic fallback, letting you shift traffic between H100 and H200 pools without client changes.

Ecosystem and limits

What works today

All major inference servers ship H200 profiles. Power envelope is similar (700W SXM). Cooling and rack specs match H100, so datacenter retrofits are trivial.

Hard ceilings

Neither GPU changes the fundamental 405B compute requirement for prefill. For ultra-long context (32K+), even H200’s 141 GB per GPU fills with KV cache at moderate batch, forcing tensor parallel + context parallel. The H200 is not a silver bullet; it pushes the wall outward.

Head-to-head comparison

Which to choose

Max throughput per dollar at scale

If you already run FP8 pipelines and have InfiniBand, 16x H100 clusters cost less upfront and deliver competitive Llama 3.1 405B H100 vs H200 throughput for prefill-heavy jobs. Choose H100.

Lowest latency single request

For a single stream, H100 FP8 decode is faster per token due to smaller weight footprint. Use H100 with aggressive FP8.

Mixed batch serving (production API)

H200’s capacity keeps more sessions resident. If you serve many concurrent users with 2–8K context, H200 single-node reduces tail latency and ops burden. Choose H200.

On-prem vs cloud

On-prem with fixed capex: H200 reduces node count and power distribution. Cloud burst: start on H100 spot, fall back to H200 when capacity tight.

Pick based on quantization tolerance and batch profile, not on headline bandwidth alone.