Running a FP8 vs INT8 H100 benchmark on production inference stacks reveals two quantization paths with identical peak tensor math but very different operational profiles. This head-to-head compares them across capabilities, cost, latency, and ecosystem so you can pick without guesswork.
Compute characteristics and peak throughput
NVIDIA’s Hopper tensor cores expose the same peak rate for 8-bit formats: 1979 TFLOPS dense (3958 with sparsity) for both FP8 and INT8 on H100 SXM. PCIe variants scale that down proportionally. The raw math ceiling is not the differentiator.
Realized throughput depends on kernel maturity and data movement. FP8 math operates directly on floating-point tensors, so many elementwise ops stay in the same dtype. INT8 convolutions and matmuls often require surrounding dequant/requant stages that eat a few percent of the budget.
Memory footprint and bandwidth
Both formats cut FP16 weight memory in half. For a 70B parameter model, FP16 weights need ~140 GB; FP8 or INT8 bring that to ~70 GB, fitting a single H100 80GB with headroom for KV cache. Memory bandwidth pressure drops similarly because the tensor cores can issue fewer loads per MAC.
The difference is in the KV cache. FP8 KV cache retains relative magnitude of activations, while INT8 KV cache needs scaling factors per block. In TensorRT-LLM you can enable FP8 KV directly:
{
"quantization": {
"fp8": { "kv_cache": true }
}
}INT8 KV cache is possible but rarely worth the calibration complexity for transformer decoders.
Accuracy and model coverage
FP8 uses an exponent, so it represents both tiny gradients and large activations without manual range selection. The E4M3 variant (max ~448) covers most LLM weight distributions; E5M2 trades precision for range. INT8 maps a linear range onto [-127, 127] via a single scale per tensor or channel.
In practice, FP8 inference on Llama-2/3 and Mistral shows <1% perplexity drift from BF16 when you cast weights and activations with a short calibration set. INT8 without quantization-aware training can lose 2–5% on tasks with outlier activations unless you use per-channel scaling and smoothquant-style tricks.
Calibration overhead
FP8 needs a few hundred forward passes to pick scaling factors for activations, but frameworks automate this. INT8 demands explicit calibration, often per-layer, and sensitive models need fine-tuning.
import torch
x = torch.randn(1024, 1024, device="cuda")
# FP8 cast – exponent handles range
x_fp8 = x.to(torch.float8_e4m3fn)
# INT8 – must pick scale manually
scale = x.abs().max() / 127
x_int8 = torch.quantize_per_tensor(x, scale, 0, torch.qint8)The INT8 snippet shows the extra step that bites every new model version.
Ergonomics and tooling
FP8 support landed in PyTorch 2.1 (torch.float8_e4m3fn), TensorRT-LLM, and vLLM (experimental for Hopper). You point the engine at the dtype and it works. INT8 has years of maturity in TensorRT, ONNX Runtime, and torch.quantize, but the API surface is heavier.
For a team shipping weekly model updates, FP8’s “cast and go” loop reduces regression risk. INT8 pipelines need recalibration scripts and signed graphs.
Cost model
The silicon cost is identical: same H100 hour, same power envelope. FP8’s slight kernel efficiency and lower calibration labor translate to lower engineering cost per deployment. INT8 can be cheaper if you already own the calibration infrastructure and target non-Hopper GPUs (Ampere supports INT8, not FP8).
If you front your H100 fleet with a gateway that honors client routing directives—n4n.ai, for instance, pins to FP8 pools and falls back to INT8 on degraded nodes—you can mix both without code changes.
Ecosystem maturity
INT8 wins on breadth: every accelerator from edge TPUs to datacenter GPUs runs it. FP8 is Hopper-forward, with MI300X and future intrinsics adopting the same 8-bit float spec. For pure H100 clusters, the FP8 ecosystem is now production-grade for the major open weights.
Limits and failure modes
FP8’s E4M3 format saturates at 448; layers with extreme outliers need per-tensor scaling or they overflow to inf. INT8’s fixed point silently clips, causing accuracy cliffs on attention maps. Neither format helps if your model is memory-bound on KV cache rather than compute-bound.
Head-to-head comparison
| Dimension | FP8 | INT8 |
|---|---|---|
| Peak throughput (H100 SXM dense) | 1979 TFLOPS | 1979 TOPS |
| Weight representation | 8-bit float (E4M3/E5M2) | 8-bit integer |
| Calibration needed | Minimal (auto range) | Required (per-tensor/channel) |
| Framework maturity | Emerging (TRT-LLM, vLLM, PyTorch 2.1+) | Mature (TensorRT, ONNX, torch.quant) |
| Memory vs FP16 | 2x reduction | 2x reduction |
| Outlier handling | Better (exponent) | Poor without smoothquant |
| Typical deployment | LLM inference on Hopper | Vision, edge, legacy serving |
Which to choose
LLM serving on H100 only: Use FP8. You get near-BF16 accuracy, half memory, and skip calibration. The kernel path is stable in TensorRT-LLM and vLLM for the common decoders.
Mixed fleet including Ampere or edge: Use INT8. FP8 won’t run on A100. If you already have INT8 calibration in CI, the marginal win from FP8 doesn’t justify porting.
Latency-critical vision or CNN inference: INT8 is battle-tested and often faster on non-Hopper due to mature fused kernels.
Rapid model rotation (new weights weekly): FP8 removes the recalibration tax. Ship the cast weights; move on.
Maximum throughput on a fixed H100 budget: They tie on paper, but FP8’s cleaner data path usually edges out by a few percent in end-to-end tokens/sec because requant overhead vanishes. Benchmark your exact model before committing.