n4nAI
Open console →
Benchmarks & performanceGuideGPU Inference Benchmarks

Choosing a GPU for LLM inference: H100 vs H200 vs B200

Practical guide to selecting the best GPU for LLM inference H100 H200 B200: compare VRAM, bandwidth, and real-world tradeoffs for production.

n4n Team4 min read916 words

Audio narration

Coming soon — every post will get a voice note here.

Picking the best GPU for LLM inference H100 H200 B200 isn’t about who has the biggest FLOPS; it’s about matching memory capacity, bandwidth, and interconnect to your model size and latency budget. This guide gives an ordered path to choose without buying the wrong silicon.

1. Define your workload constraints

Before looking at datasheets, write down three numbers: max model parameters (and quantization), max context length per request, and target requests per second (RPS). LLM inference is decode-bound; you move weights and KV cache every token, so memory bandwidth dominates steady-state throughput.

A 7B model in FP16 needs ~14GB just for weights. Add KV cache and activation overhead and you’re at 20GB+. A 70B model in FP8 still needs ~70GB. Sketch the math:

def vram_for_weights(params_b, bits=16):
    bytes_per = bits / 8
    return params_b * 1e9 * bytes_per / 1e9  # GB

print(vram_for_weights(70, bits=8))  # 70 GB

If you serve many concurrent sequences, KV cache grows linearly with batch size and context. Ignore it and you’ll OOM on the first traffic spike. Latency-sensitive chat needs low time-to-first-token; bulk extraction cares about aggregate tokens/sec.

2. Know the raw specs that matter

The three GPUs share the same CUDA ecosystem but differ sharply in memory subsystem.

H100 (SXM or PCIe)

80GB HBM3, ~3.35 TB/s bandwidth on SXM. FP8 tensor throughput around 1979 TFLOPS. Still the default workhorse because it’s available and well-supported across vLLM, TensorRT-LLM, and Hugging Face TGI.

H200 (SXM)

141GB HBM3e, ~4.8 TB/s bandwidth. Compute is identical to H100. The win is purely memory capacity and bandwidth, which directly lifts large-model and long-context throughput.

B200 (Blackwell)

192GB HBM3e, ~8 TB/s bandwidth. Adds FP4 and improved FP8, but framework support lags at launch. Use it when you need the largest single-GPU footprint or extreme bandwidth for huge batches.

{
  "h100_sxm": {"vram_gb": 80, "bw_tbs": 3.35, "fp8_tflops": 1979},
  "h200_sxm": {"vram_gb": 141, "bw_tbs": 4.8, "fp8_tflops": 1979},
  "b200": {"vram_gb": 192, "bw_tbs": 8.0, "fp4_tflops": 4500}
}

Pitfall: PCIe variants of H100 cut bandwidth and lack NVLink. Always specify SXM for inference clusters. A PCIe H100 is not the same machine as an SXM H100.

3. Map model memory footprint to GPU VRAM

Use the sum of weights, KV cache, and framework overhead. For a transformer with L layers, hidden h, context c, batch b, precision p bytes per element:

KV_GB = 2 * L * h * c * b * p / 1e9

A 70B model (L=80, h=8192) in FP16 with c=4096, b=32:

L, h, c, b, p = 80, 8192, 4096, 32, 2
kv_gb = 2 * L * h * c * b * p / 1e9
weights_gb = 70 * 1e9 * 2 / 1e9
print(round(weights_gb + kv_gb, 1))  # ~138.4 GB

That fits a single H200 (141GB) but not H100. B200 gives headroom for larger batch or context. If you quantize to FP8, weights drop to 70GB, KV to 69GB (p=1), total ~139GB still tight on H200. INT4 weights (35GB) plus KV fits H100 comfortably.

Don’t forget framework overhead: vLLM allocates ~10–20% for fragmentation and CUDA context. Add that before choosing.

4. Evaluate throughput and latency tradeoffs

Throughput scales with bandwidth when compute is saturated. For large batches, H200 delivers ~1.4x the tokens/sec of H100 at same compute because of extra bandwidth. B200 can double that again for bandwidth-bound decode.

Latency for a single small batch is less sensitive. H100 already returns first token in tens of milliseconds; H200 won’t feel faster for one user. Don’t pay for B200 if you serve low-QPS chat with short context.

Continuous batching changes the equation: with high multiplexing, the GPU is never idle, so bandwidth advantage compounds. Run a representative load before concluding.

Common trap: assuming FP8 or FP4 speedups apply to your stack. vLLM and TensorRT-LLM support H100 FP8 well; B200 FP4 needs recent builds and may break custom kernels. Validate numerics.

5. Consider cluster and interconnect

Single GPU rarely suffices beyond 70B. Eight-GPU nodes with NVLink give all-reduce bandwidth up to 900 GB/s on H100/H200. B200 NVLink is faster. Without NVLink, PCIe 5.0 x16 caps at ~64 GB/s, killing tensor parallel efficiency.

# Check NVLink status on Linux
nvidia-smi nvlink --status

If you can’t fill a NVLink node, prefer fewer large GPUs (H200/B200) over many H100s to reduce communication. NUMA placement matters: bind processes to the right socket with numactl.

6. Cost and availability reality

H100 is cheapest per hour in most clouds due to scale. H200 carries a premium for memory. B200 is scarce and pricey. Calculate $/million tokens:

def cost_per_mtok(gpu_hr, tok_per_hr):
    return gpu_hr / (tok_per_hr / 1e6)

# Illustrative only, not a benchmark
print(cost_per_mtok(2.0, 50000))  # $0.04 per M tokens

Avoid over-provisioning: a wrong B200 cluster can 10x your bill versus H100 with quantization. Use spot instances for batch jobs; reserve SXM nodes for latency-critical paths.

7. Ordered decision path

Follow this sequence:

  1. Compute total VRAM need (weights + KV + 20% headroom).
  2. If <= 70GB and low latency: pick H100.
  3. If 70–140GB or long context high batch: pick H200.
  4. If >140GB or need max bandwidth per GPU: pick B200.
  5. If multi-GPU, require NVLink SXM.
  6. Validate with a 10-minute load test using your real prompt distribution.
def choose_gpu(vram_need, latency_sensitive):
    if vram_need <= 70 and latency_sensitive:
        return "H100"
    if 70 < vram_need <= 140:
        return "H200"
    return "B200"

print(choose_gpu(138, False))  # H200

8. Validate with a real load test

Spin up the candidate GPU and run your own traffic. A minimal vLLM server:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9

Then fire concurrent requests with the OpenAI client:

from openai import OpenAI
import concurrent.futures

client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")
prompts = ["Explain PCIe vs NVLink"] * 64

def call(p):
    return client.chat.completions.create(
        model="meta-llama/Llama-3-70B-Instruct",
        messages=[{"role": "user", "content": p}],
        max_tokens=128)

with concurrent.futures.ThreadPoolExecutor(max_workers=64) as ex:
    list(ex.map(call, prompts))

Measure p50/p99 latency and total tokens/sec. If H200 meets SLA at 70% cost of B200, ship H200.

9. If you’d rather not own hardware

Self-hosting GPUs means managing drivers, cooling, and depreciation. An inference gateway like n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint, honors client routing directives, and auto-falls back when a provider is degraded. You call the model; the gateway picks the backing GPU. That removes the selection burden while keeping per-token metering.

Common pitfalls

  • Underestimating KV cache growth with context length.
  • Buying PCIe GPUs for tensor parallel workloads.
  • Chasing FP4 on B200 before your framework supports it.
  • Ignoring tail latency under batch overflow.
  • Forgetting framework overhead when summing VRAM.

Pick the smallest GPU that fits the math, then scale out only when single-device latency bounds you. That’s the disciplined path to choosing the best GPU for LLM inference H100 H200 B200.

Tagsh100h200b200gpu-benchmarkbuying-guide

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceH100 vs H200 vs B200: inference latency comparedPractical comparison of H100 vs H200 vs B200 inference latency across memory, cost, throughput, and ops, with a verdict for production LLM serving.Benchmarks & performanceB200 vs H100: tokens per second on 70B modelsB200 vs H100 tokens per second 70B: head-to-head comparison of throughput, memory, cost, and ergonomics for serving 70B LLMs in production.Benchmarks & performanceB200 NVLink and its impact on large model inferenceAnalyzing how B200 NVLink bandwidth reshapes large model inference: intra-node tensor parallelism, latency tradeoffs, and when the hardware actually pays off.Benchmarks & performanceH100 SXM vs PCIe: inference latency differencesPractical comparison of H100 SXM vs PCIe inference latency differences: raw compute, NVLink, thermals, and cost tradeoffs for LLM serving.Benchmarks & performanceH200's extra memory bandwidth: does it cut latency?Does the H200's extra memory bandwidth cut inference latency? We analyze H200 memory bandwidth inference latency versus H100 for LLM decode, prefill, and batch tradeoffs.Benchmarks & performanceLlama 3.1 405B inference: H100 vs H200 throughputA head-to-head engineering comparison of Llama 3.1 405B H100 vs H200 throughput, cost, and ergonomics to guide GPU selection for production inference.

All gpu inference benchmarks posts →