Serving a 400B-class MoE like Llama 4 Maverick in production forces a hard trade between cost and latency, and FP8 is the obvious lever. This analysis breaks down expected Llama 4 Maverick FP8 inference speed gains over BF16 by looking at the H100 roofline, quantization mechanics, and a reproducible benchmarking harness you can run today.
The hardware roofline for large MoE models
Llama 4 Maverick is a mixture-of-experts model: a fraction of its parameters is active per token, but the full weight set still resides in GPU memory. That makes the prefill phase compute-bound (you multiply activations by the active expert weights) and the decode phase memory-bound (you stream weights and KV cache from HBM for every generated token).
On an H100 SXM, the relevant numbers are public: 3.35 TB/s of HBM3 bandwidth, 989 TFLOPS of BF16 tensor core throughput, and 1979 TFLOPS of FP8 tensor core throughput. FP8 does two things at once—it halves the byte size of weights (and often activations), and it doubles raw math rate. The theoretical speedup is therefore bounded by 2x on both axes, but only if the kernel is perfectly optimized and the workload sits squarely on the limiting resource.
For decode-heavy chat traffic at batch 1, you are memory-bound. Halving weight bytes from BF16 to FP8 cuts the data moved per token roughly in half, so you approach a 2x token-rate improvement. For high-concurrency prefill, you are compute-bound, and the doubled FLOPS give you up to 2x more throughput. Real kernels incur scaling overhead, so expect 1.3–1.8x in practice.
What FP8 actually quantizes
FP8 is not a single format. NVIDIA GPUs support E4M3 (4 exponent, 3 mantissa) for forward pass and E5M2 for gradients. Inference uses E4M3 for weights and activations. The dominant serving path is W8A8—both weights and activations are cast to FP8 with per-tensor or per-channel scaling factors stored in FP32.
A minimal calibration step is required. You pass a small representative dataset through the model to compute static scales:
# pseudo-calibration with a torch-like API
scales = {}
for name, param in model.named_parameters():
if "weight" in name:
scales[name] = param.abs().max() / 448.0 # E4M3 maxvLLM and TensorRT-LLM handle this internally when you pass --quantization fp8; they either use a cached scale file or run a quick online calibration. The risk is outlier channels in large transformers—Llama architectures use RMSNorm and SwiGLU, which are tolerant, but long-context attention logits can clip.
Benchmarking harness
To measure Llama 4 Maverick FP8 inference speed, stand up a quantized server and hammer it with a realistic prompt mix. Below is a minimal vLLM launch and a Python client that records latency.
docker run --gpus all -p 8000:8000 vllm/vllm:latest \
--model meta-llama/Llama-4-Maverick-FP8 \
--quantization fp8 \
--tensor-parallel-size 8 \
--max-model-len 32768import openai, time, statistics
client = openai.Client(base_url="http://localhost:8000/v1", api_key="empty")
prompts = ["Summarize the tradeoffs of FP8 serving."] * 100
def measure(p):
t0 = time.perf_counter()
client.chat.completions.create(
model="meta-llama/Llama-4-Maverick-FP8",
messages=[{"role": "user", "content": p}],
max_tokens=256,
)
return time.perf_counter() - t0
samples = [measure(p) for p in prompts]
print(f"median {statistics.median(samples):.3f}s p99 {statistics.quantiles(samples, n=100)[-1]:.3f}s")Run the same container without --quantization fp8 (loading the BF16 checkpoint) to get a direct comparison. Keep batch size, max tokens, and concurrency identical. For throughput, use the --request-rate flag in vllm bench serve rather than a naive loop.
Expected Llama 4 Maverick FP8 inference speed numbers
Because we cannot publish exact figures for a model we did not independently silicon-benchmark here, the defensible claim is derived from the roofline and published H100 FP8 data for analogous MoE and dense models:
- Batch 1 decode: 1.4–1.6x higher output tokens/sec versus BF16, limited by HBM traffic reduction.
- Batch 32–64 prefill: 1.7–1.9x higher input tokens/sec, approaching the 2x FLOPS doubling.
- Time-to-first-token: drops proportionally to weight fetch size; expect 1.3x faster at low concurrency.
The MoE structure means active parameter count per token is small, so the compute-bound ceiling is reached at lower batch than a dense model of equal total size. That makes FP8 even more attractive: you are more often memory-bound, and weight halving is the dominant win.
Accuracy tradeoffs and when to avoid FP8
FP8 is not lossless. For Llama 4 Maverick, the published calibration recipes keep MMLU-style drops under 1% on BF16 baselines, but we have seen larger drift on function-calling schemas and precise numeric reasoning. If your application is a code generator with strict type fidelity or a financial extractor, run a held-out eval before flipping the switch.
Mitigations:
- Use per-channel weight scales, not per-tensor.
- Keep the attention softmax in BF16 (most kernels already do).
- Store KV cache in BF16 even when weights are FP8; the cache is bandwidth-hot and precision-sensitive.
Production routing and measurement
In a multi-provider setup, you rarely control the quantization flag directly—the endpoint owner does. If you want to compare Llama 4 Maverick FP8 inference speed across vendors without provisioning your own H100s, an OpenAI-compatible gateway such as n4n.ai can route to FP8-enabled backends, honor client cache-control hints to reuse prefill, and meter per-token usage so the experiment is attributable. That lets you A/B latency at equal prompt sets while the provider handles the kernel details.
When you issue the request, forward a routing directive if the gateway supports it:
{
"model": "llama-4-maverick",
"route": { "prefer": "fp8", "fallback": "bf16" },
"messages": [{ "role": "user", "content": "Benchmark prompt" }]
}The gateway should return usage with separate input/output token counts, letting you compute cost per 1k tokens at each precision.
Takeaway
FP8 is the correct default for serving Llama 4 Maverick at scale. The halved memory footprint directly attacks the decode bottleneck that dominates chat workloads, and the doubled tensor core rate covers high-concurrency prefill. You should expect a 1.4–1.8x real-world speedup over BF16 on H100-class hardware, with the high end materializing as batch size grows.
The cost is calibration and a small accuracy risk on precision-sensitive tasks. Build the harness above, run your own eval set, and if accuracy holds, ship FP8. If you cannot measure on owned hardware, use a gateway that exposes FP8 routes and per-token metering to get the same data from the cloud.