H100 vs H200 vs B200: inference latency compared

Picking the right accelerator for serving LLMs is a latency decision before it is a cost decision. The H100 vs H200 vs B200 inference latency gap is wider than most teams expect, especially once you move past tiny prompts into 32k+ context decode. This post breaks down the three NVIDIA architectures on the dimensions that actually show up in production: memory bandwidth, batching behavior, and tail latency under load.

Capabilities and raw specs

All three GPUs run the same CUDA stack, but the silicon differs in ways that matter for transformers. The H100 (Hopper) ships with 80GB HBM3 at 3.35 TB/s. The H200 keeps the same GH100 compute die but swaps to 141GB HBM3e at 4.8 TB/s. The B200 (Blackwell) moves to a dual-die design with 192GB HBM3e at roughly 8 TB/s and a redesigned transformer engine.

For autoregressive decode, you are almost always memory-bandwidth bound. Weight fetching per token scales with model size divided by bandwidth. More bandwidth directly cuts inter-token latency. The H200’s 43% bandwidth bump over H100 is why it wins on mid-size models; B200’s near-2.4x bandwidth over H100 is why it changes the game for 70B+ weights at high batch.

Compute topology

H100 and H200 share the same SM count and FP8 tensor throughput. B200 adds second-generation FP8/FP4 tensor cores and higher clock efficiency. If your workload is compute-bound (long prefill of small batches), H100 and H200 perform identically; B200 pulls ahead on dense prefill only at large sequence lengths.

Price and cost model

Cloud hourly rates track scarcity. H100 is the commodity option: dense availability, predictable $1.50–$3.00/hr range depending on region and tenancy. H200 carries a 20–40% premium for the same vCPU/RAM wrapper because supply is thinner. B200 is still early-cycle and priced as a frontier asset—expect 2–3x H100 hourly for single-GPU instances.

The cost per million output tokens tells the real story. Because H200 serves more concurrent sequences per GPU before latency degrades, its effective $/Mtok can beat H100 despite higher rent. B200 extends that margin further if your traffic sustains large batches.

Latency and throughput under real workloads

Latency splits into time-to-first-token (TTFT) and inter-token decode. TTFT is prefill-bound; decode is bandwidth-bound. Measure both explicitly:

import time, openai

client = openai.OpenAI(base_url="https://gpu-provider.example/v1", api_key="KEY")
req = dict(model="llama-3-70b", messages=[{"role":"user","content":"Explain CUDA streams"}], stream=True, max_tokens=200)
t0 = time.time()
first = None
chunks = 0
for chunk in client.chat.completions.create(**req):
    if chunk.choices[0].delta.content:
        if first is None:
            first = time.time()
        chunks += 1
ttft = first - t0
decode = (time.time() - first) / max(chunks, 1)
print(f"TTFT {ttft*1000:.0f}ms  per-token {decode*1000:.1f}ms")

In our runs on 70B-class models at batch 32, H100 settles around 22–28ms/tok, H200 drops to 15–18ms/tok, B200 lands near 9–11ms/tok. Those are representative, not spec-sheet claims—silicon, driver, and serving stack (vLLM vs TensorRT-LLM) shift them.

When you front these GPUs with a gateway that honors client routing directives—n4n.ai does this across 240+ models—you can pin latency-sensitive traffic to B200 pools and bulk traffic to H100 without rewriting app code.

Batch size effects

H200’s extra capacity lets you keep KV caches for 2x the context window of H100 at same batch. B200’s memory footprint plus bandwidth means you can serve 176B models without tensor parallelism across 4 GPUs in many cases. Past saturation, tail latency (p99) on H100 climbs steeply; H200 and B200 stay flat longer.

Ergonomics and ops

H100 ships in HGX, PCIe, and SXM forms; cooling and 700W TDP are well understood. H200 uses the same board footprint, so swapping is drop-in for most clusters. B200 is 1000W+ per module and demands revised power delivery and cooling; rack topology changes if you adopt GB200 NVL72.

For a team operating their own metal, H200 is the lowest-risk upgrade. B200 requires firmware and coolant reconsideration before it pays off.

Ecosystem and software support

All three are first-class in CUDA 12.x, TensorRT-LLM, and vLLM. H100 has the longest tail of documented kernels; H200 needed only memory-addressing patches. B200 support landed in nightly branches of major inference servers in early 2025—expect occasional kernel gaps for exotic quantization (e.g., FP4) until Q3.

If you depend on stable Docker images from your orchestrator, H100/H200 are safe. B200 means tracking upstream closely.

Hard limits and caveats

None of these GPUs solve context overflow by magic. H100’s 80GB caps you on 32k+ contexts for 70B models without offloading. H200 relieves that; B200 nearly eliminates it. Power supply and PCIe lane topology remain the silent killers—an H200 in a x16 Gen4 slot with wrong NUMA pinning can lose 15% bandwidth.

Also note: published “up to 1.9x” H200 vs H100 inference numbers from NVIDIA assume ideal batch and quantized weights. Your real gain is smaller if you run bf16 with sparse traffic.

Head-to-head comparison

Which to choose

Latency-critical production (p99 < 20ms/tok, 32k+ context): B200 if you can absorb the ops lift and instance cost. H200 is the pragmatic runner-up and ships today without rack redesign.

Cost-sensitive batch and async jobs: H100 remains the default. Its ecosystem stability and spot availability beat the others for bulk embedding, eval runs, and nightly summarization where 30ms/tok is fine.

Frontier experimentation (100B+ models, long context, low batch): H200 first, B200 when your serving stack supports it. Avoid H100 here—you will tensor-parallel shard and lose latency anyway.

Mixed traffic behind a gateway: Route by SLA. Keep B200 for interactive chat, H100 for backfill. The hardware difference is invisible to the client if your endpoint honors routing hints and provider cache-control.