Why FP8 quantization is now standard for LLM serving

The move to FP8 quantization standard LLM serving is no longer experimental: it is the default deployment precision for new inference clusters built on NVIDIA H100, H200, and AMD MI300X. FP8 cuts memory traffic and tensor-core compute cost roughly in half versus FP16 while keeping perplexity and task accuracy within a percent of baseline for most models. If you are standing up a serving stack in 2025, you should assume FP8 unless you have a specific reason to drop to FP16 or INT4.

The hardware shift that forced the issue

For a decade, FP16 and BF16 were the pragmatic precision for training and inference because they matched the native width of tensor cores and avoided the overhead of mixed INT8 pipelines. That changed when NVIDIA shipped Hopper with dedicated FP8 tensor cores that deliver 2x the throughput of BF16 at the same die area. AMD followed with MI300X supporting similar formats. The silicon decision is the root cause: once the hardware gives you cheaper math and you are memory-bandwidth bound, not using FP8 means leaving half the machine idle.

LLM inference is dominated by weight loading and KV cache reads, not by raw FLOPs. A 70B model in FP16 occupies 140 GB of VRAM just for weights; in FP8 it is 70 GB, fitting on a single H100 80GB with headroom for KV cache. That alone changes the economics of serving because it collapses multi-GPU sharding requirements for mid-size models.

What FP8 actually buys you

FP8 is not a single format. The two common encodings are E4M3 (4 exponent, 3 mantissa) and E5M2 (5 exponent, 2 mantissa). E4M3 gives higher precision for weights and activations in the typical dynamic range; E5M2 handles outliers better but with less mantissa. In practice, serving stacks use E4M3 for weights and activations, while E5M2 is mostly relevant for training gradients. For inference, E4M3 is the workhorse.

The win is twofold:

• Memory bandwidth: halved weight size means fewer bytes moved from HBM to compute.
• Compute: FP8 tensor cores double matmul throughput.

A minimal illustration of the dtype in PyTorch 2.2+:

import torch
# Cast a weight tensor to FP8 E4M3
w_fp16 = torch.randn(4096, 4096, dtype=torch.float16, device="cuda")
w_fp8 = w_fp16.to(torch.float8_e4m3fn)
# Real matmul requires scaled fp8 tensors and Hopper-specific kernels

The actual kernel call requires scaling factors, which frameworks hide. The point is the dtype is stable and supported in mainline CUDA graphs.

E4M3 vs E5M2

Use E4M3 for inference weights. E5M2 is rarely used in serving because the mantissa loss hurts activation precision. Some libraries keep a small FP16 master weight for outlier channels, but the bulk stays in FP8.

Implementation reality in serving stacks

You do not hand-roll FP8 kernels for production. Use vLLM, TensorRT-LLM, or SGLang. In vLLM, FP8 weight-only quantization is a launch flag:

vllm serve meta-llama/Llama-3-70b --quantization fp8 --dtype float8_e4m3fn

This triggers the native scaling path. TensorRT-LLM uses a calibration step to compute per-tensor scales and expects a config block:

{
  "quantization": {
    "fp8": {
      "weight": { "type": "e4m3" },
      "activation": { "type": "e4m3" }
    }
  }
}

The calibration set should be representative: 512 sequences of 2k tokens from your domain beats random web text for scale estimation. HuggingFace Optimum exposes similar flags for ONNX exports, but the mature path is the C++ serving runtimes.

Weight-only vs full tensor quantization

Weight-only FP8 (W8A16) stores weights in FP8 but computes in FP16. This is safe and gives most of the memory win with negligible accuracy drop. Full FP8 (W8A8) quantizes activations too, squeezing more compute but requiring careful scaling of layernorm outputs. For most teams, start with weight-only and move to W8A8 only if throughput is the bottleneck and you have calibrated scales.

Accuracy tradeoffs and where it breaks

FP8 is not magic. The primary failure mode is outlier channels in activations. Transformers have a few dimensions with large magnitudes; compressing them to E4M3 clips or rounds badly. The standard fix is mixed precision: keep the outlier linear layers or the attention softmax in FP16.

We have seen 70B chat models lose 0.5% on MMLU under W8A8 but recover to within 0.1% after excluding the final classifier and the first layer from quantization. Small models (<7B) are more sensitive; a 3B model may drop 2-3 points on grammar tasks if naively quantized. The FP8 quantization standard LLM serving approach assumes you will profile your specific model rather than trust a global claim.

Calibration matters. A poor scale shifts the distribution and amplifies rounding. Use a small but domain-matched set. If you serve code, calibrate on code; if you serve medical text, calibrate there.

Calibration and mixed precision

A practical recipe:

1. Load FP16 baseline.
2. Run 256 representative prompts, collect activation stats per layer.
3. Set scales at 99th percentile of absolute value.
4. Keep embedding, layernorm, and output projection in FP16.

This yields near-lossless serving for 13B-70B models. For MoE models, calibrate each expert separately because their activation distributions differ.

Operational considerations in production

Beyond the math, FP8 changes how you size batches and plan fallback. Because weights are half size, you can increase batch size or sequence length before hitting VRAM limits. That improves throughput per GPU, but watch the KV cache: it remains in FP16 unless you also quantize it.

KV cache quantization

KV cache is often the memory bottleneck for long context. FP8 KV cache is supported in recent vLLM; it cuts cache size in half with minor quality impact for contexts under 32k. For longer contexts, the rounding error accumulates, so keep KV in FP16 or use INT4 with a separate scale. The FP8 quantization standard LLM serving pattern typically pairs FP8 weights with FP16 KV unless latency budgets force otherwise.

Routing and fallback

When you aggregate capacity across providers, not all endpoints expose FP8. A gateway that honors client routing directives lets you pin to FP8-capable hardware classes while automatic fallback covers transient degradation. For example, n4n.ai forwards provider cache-control hints and routes to models that support the precision you request, so a rate-limited FP8 pool does not silently downgrade to FP16 without your knowledge. That transparency matters when you benchmark cost per token.

Takeaway

FP8 quantization standard LLM serving is the right default because the hardware subsidy is too large to ignore and the accuracy cost is manageable with mixed precision. Deploy weight-only FP8 first, calibrate on real traffic, and keep sensitive layers in FP16. If you need maximum throughput on H100-class fleets, move to W8A8 with careful outlier handling. Skip FP8 only for tiny models or compliance-bound workloads that require full FP16 audit trails. The era of FP16-as-default is over; the math and the silicon have decided.