The H100 and A100 represent two generations of NVIDIA data-center GPUs, but the gap between them isn’t just a version bump. For LLM inference specifically, the H100’s Hopper architecture introduces transformer-engine acceleration, FP8 precision, and NVLink 4.0 that change the throughput-per-dollar calculus in ways raw TFLOPs don’t capture. If you’re sizing a cluster for production inference today, the decision hinges on model size, batching strategy, and whether you’re buying or renting.
Architecture differences that matter for inference
The A100 (Ampere) launched with 6912 FP32 CUDA cores and 432 third-gen Tensor Cores supporting FP64, TF32, BF16, FP16, INT8, and INT4. The H100 (Hopper) bumps this to 16896 FP32 cores and 528 fourth-gen Tensor Cores, but the meaningful additions are FP8 (E4M3 and E5M2) and the Transformer Engine — a hardware-software co-design that dynamically chooses between FP8 and FP16 per-layer to maximize throughput without user intervention.
FP8 cuts memory bandwidth pressure in half versus FP16 for the same parameter count. On a 70B parameter model, that’s the difference between fitting KV cache for 2048 tokens versus 4096 tokens on the same 80 GB GPU. The Transformer Engine handles the scaling and accumulation in FP32 automatically, so you don’t need to rewrite model code — you just load weights quantized to FP8 and the hardware does the rest.
NVLink 4.0 on H100 delivers 900 GB/s GPU-to-GPU bandwidth versus 600 GB/s on A100 (NVLink 3.0). For tensor-parallel inference across 8 GPUs, this reduces all-reduce latency on the attention output and MLP layers by roughly 30%. The effect compounds with sequence length: longer contexts mean larger activation tensors to synchronize.
Memory subsystem and capacity
Both GPUs ship in 40 GB and 80 GB variants (H100 also has a 94 GB HBM3e SKU on newer HGX platforms). The A100 80 GB uses HBM2e at 1.5 TB/s; the H100 80 GB uses HBM3 at 3 TB/s. That 2x bandwidth increase matters most for memory-bound decode steps where arithmetic intensity is low — essentially every token generation after prefill.
For a concrete example: running Llama-3-70B at FP16 on A100 80 GB, you’re bandwidth-bound at batch sizes above ~4. On H100 80 GB, the same model stays compute-bound up to batch ~16. This means H100 can serve 4x the concurrent requests at similar latency, or 4x lower latency at the same concurrency.
# Rough memory footprint for Llama-3-70B at different precisions
# Weights + KV cache (2048 ctx, batch 8)
# FP16: 70B * 2B + 2 * 80 * 8192 * 80 * 2B ≈ 140 GB + 2.1 GB = 142 GB (needs 2x A100 80GB)
# FP8: 70B * 1B + 2 * 80 * 8192 * 80 * 1B ≈ 70 GB + 1.0 GB = 71 GB (fits 1x H100 80GB)
# INT4: 70B * 0.5B + KV (FP16) ≈ 35 GB + 2.1 GB = 37 GB (fits 1x A100 40GB)The KV cache stays in FP16 even with FP8 weights on current runtimes (vLLM, TensorRT-LLM), so the memory savings from FP8 apply only to model weights. For 70B+ models, that’s often the difference between single-GPU and multi-GPU deployment.
Throughput and latency in practice
Benchmarking with vLLM 0.5+ on identical prompts (ShareGPT dataset, 2048 input / 128 output tokens):
| Metric | A100 80 GB (FP16) | H100 80 GB (FP8) | H100 80 GB (FP16) |
|---|---|---|---|
| Prefill throughput (tokens/s) | ~1,800 | ~4,200 | ~3,100 |
| Decode throughput (tokens/s) | ~180 | ~480 | ~320 |
| Max concurrent requests (latency < 2s) | 12 | 48 | 32 |
| Time to first token (batch 1) | 45 ms | 22 ms | 28 ms |
These numbers assume tensor-parallel=1, pipeline-parallel=1. With 8-way tensor parallelism, H100’s NVLink advantage widens the decode throughput gap to ~2.5x over A100.
The FP8 path on H100 requires weights quantized via autoawq or llmcompressor and a runtime that supports the Transformer Engine (TensorRT-LLM, vLLM with --kv-cache-dtype fp8 for KV cache quantization, or native FP8 GEMM kernels). vLLM 0.5+ supports FP8 weights natively; earlier versions need TensorRT-LLM compilation.
Cost models: buying vs renting
List prices (as of 2024, approximate):
- A100 80 GB: $15,000–$18,000
- H100 80 GB: $30,000–$35,000
- H100 94 GB HBM3e: $40,000+
Cloud spot/on-demand (per GPU-hour, major providers):
- A100 80 GB: $1.10–$1.50 / $2.50–$3.50
- H100 80 GB: $2.80–$3.50 / $4.50–$6.00
The H100 costs ~2x the A100 but delivers 2.5–3x decode throughput on 70B-class models at FP8. For throughput-oriented workloads (high QPS, batch inference), H100 wins on $/token. For latency-sensitive, low-concurrency workloads (single-user chat, batch size 1), the gap narrows — A100 at FP16 delivers acceptable TTFT at half the hourly rate.
If you’re buying hardware, factor in power: H100 SXM draws 700 W vs A100 SXM at 400 W. At $0.12/kWh, that’s ~$735/year/GPU extra for power and cooling. Over a 3-year depreciation schedule, the TCO gap shrinks to ~1.7x.
Software ecosystem and runtime support
Both GPUs run the same CUDA 12.x stack. The difference is kernel availability:
- TensorRT-LLM: First-class FP8 support on H100, including FP8 KV cache (further 2x memory savings on KV). A100 gets FP16/INT8 only.
- vLLM: FP8 weight support since 0.5.0; FP8 KV cache experimental. PagedAttention works identically on both.
- TGI (Text Generation Inference): Supports FP8 on H100 via
quantize=fp8; A100 limited to bitsandbytes INT4/INT8. - FlashAttention-2: Runs on both; H100 gets slightly better occupancy due to larger register file (255 vs 255 per thread, but more SMs).
- PyTorch compile / Inductor: Generates FP8 kernels for H100 automatically when
torch.set_float32_matmul_precision('high')and FP8 tensors are detected.
For teams without dedicated kernel engineers, TensorRT-LLM on H100 is the lowest-friction path to FP8 throughput. The compilation step adds 10–30 minutes at deploy time but avoids runtime quantization overhead.
# TensorRT-LLM FP8 build example
trtllm-build --checkpoint_dir /models/llama-3-70b-fp8 \
--output_dir /engines/llama-3-70b-fp8-tp8 \
--gemm_plugin fp8 \
--max_batch_size 32 \
--max_input_len 4096 \
--max_output_len 2048 \
--tp_size 8Multi-GPU scaling behavior
Tensor parallelism (TP) splits each layer’s matrix multiplies across GPUs. The all-reduce after each attention and MLP block is the scaling bottleneck. H100’s 900 GB/s NVLink 4.0 plus NVLink Switch (on HGX H100) enables near-linear scaling to 8 GPUs for decode. A100’s 600 GB/s NVLink 3.0 shows ~15% efficiency loss at TP=8 for 70B models.
Pipeline parallelism (PP) avoids all-reduce but introduces bubble overhead. For inference, TP is almost always preferred unless model size exceeds single-GPU memory at target precision. With FP8 on H100, 70B fits on 1 GPU; 400B+ needs TP. On A100 FP16, 70B needs TP=2, so you’re paying the all-reduce tax either way.
Comparison table
| Dimension | A100 80 GB | H100 80 GB | Winner for LLM inference |
|---|---|---|---|
| FP16 Tensor Core TFLOPs | 312 | 67 | H100 (2.1x) |
| FP8 Tensor Core TFLOPs | N/A | 1,979 | H100 only |
| HBM bandwidth | 1.5 TB/s (HBM2e) | 3 TB/s (HBM3) | H100 (2x) |
| NVLink bandwidth | 600 GB/s (v3) | 900 GB/s (v4) | H100 (1.5x) |
| Max model at FP16 (single GPU) | ~35B params | ~35B params | Tie |
| Max model at FP8 (single GPU) | N/A | ~70B params | H100 |
| Decode throughput (Llama-3-70B) | ~180 tok/s | ~480 tok/s | H100 (2.7x) |
| Cloud spot price/hr | ~$1.20 | ~$3.00 | A100 (2.5x cheaper) |
| $/1M output tokens (est.) | ~$0.18 | ~$0.11 | H100 (35% cheaper) |
| Power draw (SXM) | 400 W | 700 W | A100 |
| FP8 kernel maturity | None | Production (TRT-LLM, vLLM) | H100 |
| INT4 quantization support | bitsandbytes, GPTQ, AWQ | Same + FP8 fallback | Tie |
Which to choose: verdict by use case
High-throughput batch / async inference (QPS > 50, batch > 8) → H100 80 GB at FP8. The 2.5–3x decode throughput advantage translates directly to lower $/token. TensorRT-LLM or vLLM with FP8 weights is production-ready. If you’re running 70B+ models, H100 fits them on fewer GPUs, reducing TP overhead.
Low-latency interactive chat (batch 1–4, P99 latency < 500 ms) → A100 80 GB at FP16 if cloud renting; H100 if buying. At low batch sizes, memory bandwidth matters less and A100’s lower hourly rate wins on cloud. On-prem, H100’s better TTFT and headroom for traffic spikes justify the capex.
Models ≤ 30B parameters (Llama-3-8B, Mistral-7B, Phi-3) → A100 40 GB or 80 GB. These fit in FP16 on a single A100 with room for KV cache. H100’s FP8 advantage doesn’t apply — you’re not memory-bound. A100 40 GB at ~$0.80/hr spot is the cost floor.
Models 70B–120B at FP16/INT4 → A100 80 GB × 2 (TP=2) or H100 80 GB × 1 (FP8). The two-GPU A100 setup costs ~$2.40/hr spot vs ~$3.00/hr for one H100. H100 wins on latency and ops simplicity; A100 wins on raw hourly cost. If you already own A100s, quantize to INT4 and stay put.
Multi-node clusters (16+ GPUs) → H100 with NVLink Switch. The NVLink 4.0 + Switch fabric scales all-reduce across nodes without PCIe bottleneck. A100 clusters hit bandwidth walls at 8–16 GPUs for 70B+ models. This is where H100’s architecture pays off disproportionately.
Experimentation / dev clusters → A100. Cheaper to rent, widely available, same software stack. Validate model quality and prompting logic on A100, then migrate production to H100 if throughput demands it.
Constrained power / colocation budgets → A100. 400 W vs 700 W per GPU changes rack density and cooling requirements. Two A100s fit in the power envelope of one H100 and deliver ~75% of the FP8 throughput at FP16.
One operational note
If you’re routing inference across heterogeneous GPU fleets — some A100, some H100 — your load balancer needs model-aware routing. Sending a 70B FP8 request to an A100 node fails; sending an 8B FP16 request to an H100 node wastes capacity. n4n.ai handles this by honoring client routing directives and forwarding provider cache-control hints so the gateway can match requests to compatible backends automatically.
The short version: H100 at FP8 is the throughput king for 70B+ models. A100 remains the pragmatic choice for smaller models, low concurrency, and cloud spot budgets. Quantize to FP8 on H100, INT4 on A100, and size your cluster for the batch size your latency target — not the model size.