Choosing between FP8 and BF16 for serving Llama 3.1 is no longer a theoretical exercise—it directly determines your GPU bill and tail latency. This head-to-head on FP8 vs BF16 inference speed breaks down what each numeric format costs you in throughput, memory, and engineering effort on H100-class hardware.
Numeric formats in one paragraph
BF16 stores 1 sign, 8 exponent, 7 mantissa bits. It covers the same dynamic range as FP32, which is why it became the default training and inference dtype for large transformers. FP8 comes in two flavors: E4M3 (4 exponent, 3 mantissa) for tensors, and E5M2 for gradients. On NVIDIA Hopper and Blackwell, FP8 Tensor Cores execute multiply-accumulate at double the rate of BF16 for the same shape. Llama 3.1 ships with checkpoint conversions that map cleanly to E4M3 weight-only or weight-activation quantization.
Test setup we assume
We are not publishing fresh numbers here; we describe the regime that determines results you will reproduce.
Hardware
H100 SXM 80GB or equivalent with FP8 support. On A100, FP8 is emulated and loses all advantage—do not bother.
Software
vLLM 0.6.x or TensorRT-LLM 0.12+ with --dtype fp8 for the FP8 path. BF16 path is the same engine without the flag. Llama 3.1 8B, 70B, and 405B all load via meta-llama/Llama-3.1-* on HuggingFace.
Head-to-head dimensions
The following table summarizes the six axes that matter for a production rollout.
| Dimension | FP8 | BF16 |
|---|---|---|
| Capabilities | E4M3 8-bit float; Llama 3.1 weights convertible with <0.1 perplexity delta on 70B/405B | 16-bit float, native training format, reference accuracy |
| Price/cost model | ~50% lower VRAM, fits larger batch on same GPU; cheaper per-token on FP8-capable instances | Higher memory footprint, may require 2x GPUs for same batch |
| Latency/throughput | Up to 2x matmul throughput on H100; real FP8 vs BF16 inference speed gain 1.2–1.7x for compute-bound batches | Baseline; memory-bandwidth bound on small batches, near-peak on large |
| Ergonomics | Requires HW support, vLLM/TRT-LLM paths; calibration optional | Works everywhere; torch_dtype=bfloat16 just works |
| Ecosystem | Limited to recent CUDA, specific serving engines; HF fp8 support maturing | Universal: PyTorch, TF, ONNX, all clouds |
| Limits | Degrades on outlier-heavy layers if poorly scaled; no CPU fallback | No speedup on non-FP8 HW; 2x memory vs FP8 |
Capabilities
BF16 preserves the full exponent range of FP32. For Llama 3.1, attention logits and layer norms stay in BF16 even in an FP8 deploy; only matmuls get compressed. FP8 E4M3 clips to ±448 with 3 mantissa bits. In practice, Llama 3.1 70B shows negligible accuracy drop on MMLU when using per-tensor scaled FP8 weights. The 405B variant benefits most because it is hardest to fit in memory.
Price/cost model
Memory is the dominant cost. Llama 3.1 70B in BF16 needs ~140GB VRAM (weights + KV cache + overhead), forcing a 2xH100 node. In FP8, weights drop to ~70GB, leaving headroom for larger batch sizes on a single card. That single fact changes the per-token economics more than raw FLOPs. Cloud hourly rates do not discount FP8, but you use half the GPUs.
Latency/throughput
This is where FP8 vs BF16 inference speed gets interesting. For a 8B model at batch 1, the kernel is memory-bound; FP8 saves bandwidth but compute is idle, so speedup is modest (often 1.1–1.3x). At batch 32 on 70B, matmuls dominate and H100’s FP8 Tensor Cores deliver closer to 1.6x more tokens/sec. Your mileage depends on sequence length and KV cache pressure.
Ergonomics
BF16 is a one-liner:
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct",
torch_dtype=torch.bfloat16,
device_map="auto"
)FP8 needs a serving engine that understands it:
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--dtype fp8 \
--tensor-parallel-size 1 \
--max-model-len 8192No calibration script is mandatory for weight-only FP8 in vLLM, but you should still run a tiny validation set to catch outlier channels.
Ecosystem
Every inference server speaks BF16. FP8 is younger: TensorRT-LLM, vLLM, and recent HuggingFace transformers nightly have it. If you rely on llama.cpp or CPU inference, FP8 is nonexistent. Plan your fallback path before committing.
Limits
FP8 cannot recover from a bad scaling factor. Llama 3.1’s attention outputs have sporadic large magnitudes; static per-tensor scaling can clip them. Use dynamic activation scaling if your engine supports it. BF16’s only hard limit is that it does nothing on hardware without fast BF16 paths (almost none today) and costs memory.
Deployment sketch
A minimal vLLM benchmark to feel the difference:
# BF16 baseline
vllm bench serve meta-llama/Llama-3.1-8B-Instruct --dtype bfloat16 \
--num-prompts 100 --request-rate 16
# FP8
vllm bench serve meta-llama/Llama-3.1-8B-Instruct --dtype fp8 \
--num-prompts 100 --request-rate 16Compare the tokens/s and p99 latency lines. On a single H100, the 8B model will show small gains; the 70B will show the gap.
If you front your fleet with a gateway that honors client routing directives, you can pin FP8-capable nodes for batch jobs while sending latency-sensitive small requests to BF16 paths when FP8 hardware is saturated. n4n.ai forwards such routing hints and falls back automatically when a provider is degraded, which keeps the format switch invisible to app code.
Which to choose
Cost-sensitive batch inference (70B/405B, offline)
Use FP8. The memory saving alone justifies it, and throughput gains compound across long jobs. Validate accuracy on your eval set once.
Low-latency interactive (8B, single user)
BF16 is fine. The FP8 vs BF16 inference speed gap at batch 1 is too small to risk format edge cases. Keep it simple.
Maximum accuracy required (eval, legal, medical)
BF16. Avoid any perplexity risk until your pipeline has proven FP8 scaling on domain data.
Mixed fleet with variable load
Run both. Route large batches to FP8 pools, spillover to BF16. Use a gateway that understands routing directives and provider cache hints so repeated prompts hit the same precision path.
FP8 is not a silver bullet, but for Llama 3.1 at scale it is the cheapest performance lever you have after batching. BF16 remains the safe default. Measure on your own prompts, then commit.