INT4 vs FP16 inference speed on Mixtral 8x7B

The debate over INT4 vs FP16 Mixtral inference speed is mostly a question of memory bandwidth versus precision. On Mixtral 8x7B, quantizing to INT4 shrinks the weight footprint from ~94GB to ~24GB, which changes the hardware you need and the throughput you get. Both formats run the same sparse expert architecture, but they impose different constraints on serving infrastructure and on the teams that operate them.

Capabilities and Accuracy

Mixtral 8x7B is a mixture-of-experts model with 46.7B total parameters and 12.9B active per token. FP16 stores each parameter as a 16-bit float, preserving the original training distribution. INT4 packs four weights into a single 16-bit word with a per-block scaling factor, trading some numerical fidelity for density.

In practice, INT4 (via GPTQ, AWQ, or GGUF Q4_K) retains the vast majority of FP16 accuracy on standard benchmarks for this architecture. The MoE routing is robust to small weight perturbations because only a subset of experts fires per token. You lose a little calibration on low-probability tokens; FP16 wins if you need maximum reproducibility for eval harnesses or if you are fine-tuning further.

# Pseudo-code for a local accuracy spot-check
from lm_eval import evaluate
fp16_score = evaluate("mistralai/Mixtral-8x7B-v0.1", dtype="fp16")["perplexity"]
int4_score = evaluate("mixtral-8x7b-gptq-int4", dtype="int4")["perplexity"]
# Typical gap on WikiText2 is <0.5 perplexity points
assert fp16_score - int4_score < 0.5

Where INT4 drifts

Math chains and long-context retrieval can expose quantization noise. If your pipeline uses Mixtral for tool calling with strict schema, validate the INT4 build against your own validation set before shipping.

Cost Model and Hardware Requirements

FP16 Mixtral requires at least two 80GB A100s (or H100s) because 94GB of weights plus KV cache and activations exceed single-GPU memory. INT4 fits on a single 24GB consumer card (RTX 4090) for inference with modest batch sizes, or a single 80GB A100 with massive batch headroom.

That hardware gap drives cost. A cloud A100-hour runs roughly $1–2; a 4090 instance is often $0.20–0.50. Serving INT4 can cut hourly GPU cost by 4–8x for the same request volume if you are memory-bound. When you meter per-token usage through a gateway (n4n.ai provides this), INT4’s smaller footprint translates directly into lower instance cost per million tokens.

# Crude VRAM math
python -c "print(46.7e9 * 2 / 1e9, 'GB for FP16')"   # 93.4 GB
python -c "print(46.7e9 * 0.5 / 1e9, 'GB for INT4')" # 23.35 GB

Total cost of ownership

FP16 needs multi-GPU orchestration (NCCL, tensor parallel). INT4 on a single card removes distributed-serving complexity but limits max concurrency. Factor in engineering time, not just GPU stickers.

Latency and Throughput

INT4 vs FP16 Mixtral inference speed splits on the memory wall. Decoding is bandwidth-bound: each token requires reading all weights from VRAM. INT4 reads 1/4 the bytes, so on the same GPU you get proportionally higher tokens/sec until compute (dequant + matmul) saturates.

On a single 4090, INT4 serves ~40–60 tokens/sec for a single stream; FP16 cannot run there at all. On dual A100, FP16 might hit 200 tokens/sec with continuous batching; INT4 on one A100 often matches or beats that with lower power draw.

Latency per token is similar in milliseconds because dequant is cheap, but time-to-first-token improves with INT4 when the model fits entirely in cache and avoids cross-GPU links.

{
  "fp16_dual_a100": {"tokens_per_sec": 200, "ttft_ms": 45, "per_token_ms": 5},
  "int4_single_a100": {"tokens_per_sec": 220, "ttft_ms": 30, "per_token_ms": 4.5}
}

Figures above illustrate bandwidth-bound scaling, not a specific measured run.

Prefill vs decode

Prefill is compute-heavy; INT4’s advantage is smaller there because you are multiplying activations anyway. Decode is where INT4 pulls ahead. Batch many requests and the KV cache (still FP16) becomes the dominant memory user, narrowing the gap.

Ergonomics: Serving and Tooling

FP16 is the native format from Mistral AI. You point vLLM or TGI at the repo and go.

from vllm import LLM
llm = LLM(model="mistralai/Mixtral-8x7B-Instruct-v0.1", dtype="half")

INT4 needs a quantized checkpoint. You either download a community GPTQ/AWQ build or run conversion with a calibration set:

# GPTQ convert (requires calibration set)
python -m transformers.commands.quantize \
  --model mistralai/Mixtral-8x7B-v0.1 \
  --method gptq --bits 4 --output mixtral-gptq-4bit

Serving INT4 with llama.cpp is straightforward via GGUF:

./server -m mixtral-8x7b-Q4_K_M.gguf -c 4096 -ngl 40

The extra preprocessing step is the only real friction. Once baked, INT4 loads faster because fewer bytes hit the bus.

Ecosystem and Framework Support

Both formats work in vLLM, TensorRT-LLM, llama.cpp, and Hugging Face TGI. FP16 has universal kernel support; INT4 relies on quantized kernels (Marlin, ExLlamaV2, llama.cpp Q4) that are mature but occasionally lag on new GPU architectures.

If you use an OpenAI-compatible gateway such as n4n.ai, the quantization is abstracted: you request mixtral-8x7b and the backend routes to a provider running either variant, honoring your cache-control headers. Your client code stays identical.

// Same client for both variants
const res = await fetch("https://api.n4n.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${key}`,
    "Cache-Control": "max-age=300"
  },
  body: JSON.stringify({ model: "mixtral-8x7b", messages })
});

Limits and Trade-offs

INT4 has hard limits: not all fine-tunes survive 4-bit; long-context KV cache still grows linearly and is stored in FP16, so a 32k context on a 24GB card may OOM despite small weights. FP16 scales to long context more predictably but costs more per node.

FP16 cannot run on commodity hardware. INT4 may show slight degradation on math-heavy chains. Choose based on deployment target, not raw speed leaderboard stats.

Operational guardrails

Monitor perplexity on production traffic samples for INT4.
Keep a FP16 canary instance for A/B if accuracy complaints surface.
Set max batch size explicitly; INT4 hides memory headroom until KV cache spikes.

Head-to-Head Comparison

Dimension	FP16 Mixtral 8x7B	INT4 Mixtral 8x7B
Weight size	~94 GB	~24 GB
Min GPU	2x A100 80GB	1x RTX 4090 (24GB)
Relative throughput/GPU	Baseline	2–4x higher (bandwidth-bound)
Accuracy vs base	Reference	<0.5% drop typical
Serving friction	None	Quantization step required
Long-context cost	High VRAM	Lower but KV cache still FP16
Framework support	Universal	Mature but variant-specific
Multi-GPU need	Yes	No (single card viable)

Which to Choose

Prototype on a budget: INT4 on a single consumer GPU. You get Mixtral quality at 1/4 the hardware cost and enough speed for dev loops. Use llama.cpp or ExLlamaV2.

Production at scale: If you already run A100/H100 clusters, FP16 avoids quantization drift and simplifies CI. But INT4 on single A100s often yields better price-per-token; use it unless eval regressions appear.

Latency-sensitive edge: INT4 is the only option that fits in 24GB. Deploy with TensorRT-LLM or llama.cpp and tune -ngl for layer offload.

Maximum accuracy / scientific use: FP16 on dual GPUs. Keep INT4 for draft generation or request routing.

Gateway consumers: Pick the model name exposed by your provider and benchmark your own traffic. The INT4 vs FP16 Mixtral inference speed question resolves to where your bottleneck lives: memory cost or precision headroom. Match the format to your hardware, not to a generic benchmark.