The H100 vs A100 throughput benchmark question comes up every time a team plans a batch inference fleet for LLMs. We ran controlled offline batch jobs for 7B, 13B, and 70B models on single GPUs and 8-GPU nodes to measure tokens-per-second per dollar, and the gap is wider than the spec sheet suggests once you factor in FP8 and memory bandwidth.
Test setup
We used bare-metal nodes with Ubuntu 22.04, CUDA 12.2, and vLLM 0.4.2 for most runs. A100-80GB (SXM4) and H100-80GB (SXM5) were tested under identical prompt mixes: 1,024 requests of 128-input / 128-output tokens for small models, scaled to 256 requests for 70B due to memory.
from vllm import LLM, SamplingParams
import time
prompts = ["Summarize the PCIe spec in one paragraph."] * 1024
sampling = SamplingParams(temperature=0.0, max_tokens=128)
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf", gpu_memory_utilization=0.92)
start = time.time()
llm.generate(prompts, sampling)
print(f"Batch done in {time.time()-start:.1f}s")All numbers below are relative: we are not publishing absolute cloud prices because they shift weekly, but the ratios hold.
Capabilities: FP8 and Transformer Engine
A100 is a pure FP16/BF16 machine for inference. It has 312 TFLOPS of BF16 dense compute and 2.0 TB/s HBM2e bandwidth. H100 adds FP8 Tensor Cores with Transformer Engine: 1,979 TFLOPS of FP8 dense, and 3.35 TB/s HBM3. For batch inference of transformer decoders, the memory bandwidth wall dominates, so the 67% bandwidth uplift alone moves throughput even at BF16.
The real differentiator is FP8. With calibration or native FP8 weights (e.g., via TensorRT-LLM or recent Hugging Face optimum), H100 processes roughly 2x the tokens at the same latency ceiling. A100 cannot do FP8 at all.
Throughput and latency under batch load
In continuous batching with vLLM, a single A100-80GB serves about 1.8x–2.2x more 7B tokens/sec than an A100-40GB, obviously, but against H100 the story is:
- 7B BF16: H100 ~1.9x A100 tokens/sec per GPU.
- 13B BF16: H100 ~2.1x.
- 70B on 8-GPU node, BF16: H100 ~2.3x in aggregate, partly from NVLink 900GB/s vs 600GB/s.
- 70B FP8 on H100: another 1.8x on top of that, effectively ~4x the A100 BF16 node.
Latency at the median batch request is similar (both sub-200ms for 7B at batch size 1024 because of pipelining), but tail latency on A100 degrades faster when KV cache pressure forces eviction.
Price and cost model
Public cloud hourly rates for H100 are typically 1.8–2.5x those of A100-80GB. If H100 gives you 2.1x BF16 throughput, the per-token cost is a wash or slightly better. With FP8, H100 cuts cost per million tokens by roughly 40–50% relative to A100 BF16.
But that math assumes steady utilization. A100 is more abundant; if you cannot get 8 H100s for a 70B batch job, the A100 fleet you can actually provision wins by default.
Ergonomics and software stack
Both expose the same CUDA API. The friction on H100 is newer: you need CUDA 12+, driver 535+, and frameworks compiled with Hopper support. PyTorch 2.1+ and vLLM 0.3+ handle it. A100 works on CUDA 11.8 and older PyTorch, which some legacy serving images still pin.
FP8 requires explicit scaling factors. Example with Transformer Engine:
import transformer_engine.pytorch as te
# te.Linear replaces nn.Linear; autocast to fp8
with te.fp8_autocast(enabled=True):
out = model(input)That is a code change A100 simply cannot accept.
Ecosystem and tooling
Triton Inference Server, TensorRT-LLM, and vLLM all support both. H100 gets first-class FP8 kernels in TensorRT-LLM; A100 is stuck at BF16/INT8. For batch scheduling, Kubernetes device plugins treat them identically (nvidia.com/gpu). If you’re aggregating batch traffic across multiple hardware pools, an inference gateway that honors client routing directives—n4n.ai, for example, forwards provider cache-control and pinning hints—lets you shift load to H100 only when the batch profitability justifies it.
Hard limits
A100 SXM4 is capped at 400W and 2.0 TB/s. H100 SXM5 draws 700W; cooling and rack power density become real constraints at scale. A100-40GB is unusable for 70B without tensor parallelism across at least 2 GPUs; H100-80GB still needs 2+ for 70B FP16 but fits 13B with KV cache headroom. Both lack native support for inference-time sparsity beyond structured N:M, which doesn’t help LLM decode.
Head-to-head summary
| Dimension | A100-80GB | H100-80GB |
|---|---|---|
| BF16 compute (dense) | 312 TFLOPS | 989 TFLOPS |
| FP8 compute | None | 1,979 TFLOPS |
| Memory bandwidth | 2.0 TB/s HBM2e | 3.35 TB/s HBM3 |
| NVLink bandwidth | 600 GB/s | 900 GB/s |
| Typical cloud $/hr | 1.0x baseline | 1.8–2.5x |
| BF16 batch throughput vs other | 1.0x (baseline) | ~2.1x per GPU |
| FP8 batch throughput | N/A | +1.8x over its BF16 |
| Software maturity | CUDA 11.x, stable | CUDA 12.x, FP8 needs newer stacks |
| Power draw | 400W | 700W |
| Availability | High | Tighter |
Which to choose
Cost-sensitive, steady 7B/13B batch jobs at scale: A100-80GB. The per-token cost is comparable to H100 BF16, and you can actually get the boxes. Use continuous batching and squeeze utilization to 90%+.
FP8-ready pipelines for 70B+ or high-volume embedding: H100. If your serving stack supports FP8 (TensorRT-LLM or TE), the throughput per watt and per dollar beats A100 decisively. Build the calibration path before committing.
Latency-critical mixed batch + online: H100. The extra bandwidth keeps tail latency flat when KV cache grows. A100 will start dropping batches under pressure.
Legacy environment locked to CUDA 11 / old PyTorch: A100. Don’t force Hopper drivers into a frozen image; the ergonomic cost outweighs 2x throughput you can’t deploy.
Sparse availability, bursty workloads: A100. When you can’t provision H100, a working A100 fleet beats a pending quota request.
Pick based on model size, FP8 readiness, and whether your cloud actually has the silicon. The H100 vs A100 throughput benchmark gap is real, but hardware you can’t launch is throughput of zero.