n4nAI
Open console →

Topic

Quantization Impact on Inference Speed

12 posts on quantization impact on inference speed — part of benchmarks & performance on the n4n AI blog.

12 postsBenchmarks & performanceAll posts
Benchmarks & performanceAnalysis

Why FP8 quantization is now standard for LLM serving

FP8 quantization standard LLM serving emerged from hardware gains and near-lossless accuracy; we analyze the tradeoffs and why it's now the default for inference.

5 min read
Benchmarks & performanceAnalysis

Quantization impact on inference speed: DeepSeek V3

Analysis of how FP8, INT8, and INT4 quantization affect DeepSeek V3 inference speed, with memory math, quality tradeoffs, and serving recommendations.

4 min read
Benchmarks & performanceComparison

Quantization benchmark: FP8 vs INT8 on H100 GPUs

Head-to-head FP8 vs INT8 H100 real benchmark: compare throughput, accuracy, ergonomics, and cost for LLM serving, plus which to choose.

4 min read
Benchmarks & performanceComparison

Quality vs speed: quantized vs full-precision model outputs

A practical head-to-head comparison of quantized vs full-precision model quality across inference speed, cost, capabilities, ergonomics, and limits.

4 min read
Benchmarks & performanceComparison

INT4 vs FP16 inference speed on Mixtral 8x7B

Practical comparison of INT4 vs FP16 Mixtral inference speed across hardware cost, throughput, accuracy, and ergonomics for engineers.

4 min read
Benchmarks & performanceAnalysis

INT4 quantization speed gains on consumer GPUs

Analysis of INT4 quantization consumer GPU speed gains: real throughput wins on bandwidth-bound hardware, accuracy tradeoffs, and when to ship it.

4 min read
Benchmarks & performanceAnalysis

How much latency quantization saves on 70B parameter models

Quantization latency savings 70B models come from cutting memory bandwidth needs; INT8 roughly halves decode latency vs FP16 with minimal accuracy loss.

4 min read
Benchmarks & performanceComparison

GPTQ vs AWQ: quantization method speed comparison

A practical GPTQ vs AWQ speed comparison across latency, cost, and ecosystem to help engineers pick the right 4-bit LLM quant for production.

5 min read
Benchmarks & performanceAnalysis

Does quantization hurt reasoning benchmarks like GSM8K?

Quantization affects reasoning accuracy on GSM8K only marginally at 8-bit and recoverably at 4-bit; we analyze tradeoffs for production LLM serving.

5 min read
Benchmarks & performanceAnalysis

INT4 quantization: how much speed for how much accuracy

Analysis of INT4 quantization speed accuracy tradeoff: real throughput gains vs task-dependent degradation, with deployment code and a decisive recommendation.

3 min read
Benchmarks & performanceComparison

FP8 vs BF16: inference speed benchmark for Llama 3.1

Practical comparison of FP8 vs BF16 inference speed for Llama 3.1: throughput, cost, accuracy, and deployment tradeoffs on H100-class hardware.

4 min read
Benchmarks & performanceAnalysis

Benchmarking FP8 inference speed for Llama 4 Maverick

Analyze Llama 4 Maverick FP8 inference speed on H100: methodology, expected speedups vs BF16, tradeoffs, and a production benchmarking harness.

4 min read

More topics in benchmarks & performance

See all benchmarks & performance posts →