The question of whether quantization reasoning accuracy GSM8K suffers under reduced precision cuts to the core of cost-sensitive LLM deployment. Our thesis: 8-bit quantization is effectively lossless for chain-of-thought math, while 4-bit demands careful method selection but remains viable for most production traffic. Treat precision as a dial, not a switch.
What GSM8K actually stresses
GSM8K is a corpus of 8.5K grade-school math word problems requiring 2–8 step arithmetic with explicit intermediate reasoning. A model must generate a coherent chain of thought and emit a final numeric answer after a #### delimiter. The exact-match scoring means any digit error in the final line counts as a full failure, but the multi-step format gives the model multiple chances to recover from a single noisy token.
This makes it a decent proxy for multi-hop reasoning where intermediate state matters. It is not a test of factual recall or coding syntax; it is a test of whether the weight matrix can consistently route probability mass toward correct arithmetic operators and operands under sampling.
How quantization changes the math
Quantization maps FP16 weights (and sometimes activations) to lower-bit representations: INT8, INT4, NF4, or mixed schemes. Two families dominate production:
Weight-only vs joint quantization
Weight-only methods (GPTQ, AWQ, llama.cpp q4/q5) compress static parameters and dequantize on the fly in the GEMM kernel. Activation quantization (SmoothQuant, FP8) also reduces memory bandwidth for intermediates. For reasoning tasks, weight noise is the dominant factor because every token’s hidden state is a linear combination of quantized columns.
Calibration and outliers
Post-training quantization (PTQ) uses a small calibration set to estimate per-tensor or per-channel scaling factors. Transformer activations contain extreme outliers in a tiny fraction of channels. If those channels are clipped or poorly scaled, semantic signal collapses. This is why naive round-to-nearest INT4 on a 7B model can crater GSM8K while a method aware of outliers does not.
from transformers import BitsAndBytesConfig
cfg = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype="bfloat16",
)
# NF4 + double quant is the bitsandbytes baseline for 4-bit LLM inferenceEmpirical impact on quantization reasoning accuracy GSM8K
The literature and community eval harnesses converge on a clear trend: precision reduction hurts small models more than large ones. A 70B model has redundant capacity; INT4 weights still encode the solution path. A 7B model operates closer to the entropy floor, so rounding errors propagate.
AWQ’s central observation is that roughly 1% of weights act as activation outliers and dominate downstream magnitude. Protecting those channels at higher precision recovers most GSM8K accuracy. The AWQ paper demonstrates this recovers the majority of the gap versus naive INT4 on math tasks. GPTQ minimizes layer-wise reconstruction error via approximate second-order information; it works well but can lag AWQ on reasoning because it does not explicitly account for activation salience.
# Build a q4_0 GGUF from a float16 checkpoint
./llama-quantize models/llama-2-13b-f16.gguf models/llama-2-13b-q4_0.gguf q4_0
# Run GSM8K with lm-evaluation-harness
python -m lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-13b-hf,load_in_4bit=True \
--tasks gsm8k --num_fewshot 5 --batch_size 8Model size matters
For 30B+ parameter models, INT8 is indistinguishable from FP16 on GSM8K within seed variance. INT4 AWQ typically trails by a small absolute margin. For 7B–13B class models, INT4 can drop several points unless salient weighting is applied. If your SLA tolerates a minor accuracy dip for a 3x memory reduction, the trade is usually worth it.
Why reasoning is surprisingly robust
Chain-of-thought is not a single high-precision operation; it is a sequence of softmax samples. Quantization adds roughly zero-mean noise to logits. Because the model’s confidence margin on the correct next-token choice is often large, the argmax rarely flips. Errors that do flip a token tend to be in superficial phrasing, not the arithmetic core, provided the model is not near chance performance.
Self-consistency decoding (sample k chains, majority vote) closes much of the quantization reasoning accuracy GSM8K gap. The noisy chains that produce a wrong final answer are often outvoted by clean ones. In practice, running 8-way voting on a 4-bit model frequently matches FP16 greedy decoding on this benchmark.
# Sketch of self-consistency aggregation
from collections import Counter
def sc_vote(responses):
answers = [extract_final(r) for r in responses]
return Counter(answers).most_common(1)[0][0]
# extract_final pulls the string after '####' and parses digitsSpeed and memory tradeoffs
Quantization’s primary payoff is throughput per dollar. INT8 halves weight memory and improves cache locality; INT4 quarters it. A 70B FP16 model needs ~140GB VRAM (2×80GB A100). INT4 fits in a single 40GB A100. Tokens/sec per dollar scales nearly linearly with memory reduction because you can either drop a GPU or increase batch size.
{
"fp16_70b": {"gpu_mem_gb": 140, "max_batch": 8, "gpu_count": 2},
"int4_70b": {"gpu_mem_gb": 38, "max_batch": 24, "gpu_count": 1}
}The cost is not just accuracy. Dequantization overhead eats part of the speed gain if the kernel isn’t fused. Use EXL2, AWQ CUDA kernels, or llama.cpp’s fused metal/CPU paths to avoid host-side conversion stalls. Measure tokens/sec, not just VRAM, before committing.
Deployment patterns
In a gateway serving many model variants, quantization is a routing lever. When a full-precision endpoint is rate-limited or degraded, falling back to a 4-bit replica keeps p99 latency bounded at the expense of a small accuracy regression. n4n.ai exposes this by honoring client routing directives across 240+ models behind one OpenAI-compatible endpoint, so a caller can pin quality=high or cost=low without rewriting prompts or juggling provider SDKs.
This matters because GSM8K-style traffic in production is rarely isolated. You might serve a mix of summarization, extraction, and math. Keeping both FP16 and INT4 weights warm lets the gateway shed load dynamically.
How to measure the gap correctly
Do not trust a single greedy run. GSM8K variance across seeds and few-shot orders is real. Use the lm-eval-harness with fixed few-shot and at least 3 seeds. Diff the failure sets:
# Compare FP16 vs INT4 failure indices
fp16_fail = set(load_results("fp16.json")["failures"])
int4_fail = set(load_results("int4.json")["failures"])
print("new int4 failures:", int4_fail - fp16_fail)
print("recovered by int4:", fp16_fail - int4_fail)If the symmetric difference is small and the new failures are arithmetic typos, quantization is safe. If INT4 introduces conceptual errors (wrong operation selection), the model is too small for that precision.
When not to quantize
If your application demands exact arithmetic on long chains—financial reconciliation, symbolic proof, constraint solving—avoid sub-8-bit. Use tool augmentation: let the LLM emit a Python snippet and execute it. Quantization noise becomes irrelevant when the model delegates calculation to a deterministic interpreter.
Also, if you fine-tune on domain math, re-calibrate quantization after training. A model adapted to novel notation shifts its outlier distribution; stale scales silently degrade GSM8K-style tasks. Always re-run PTQ on the final weights.
Takeaway
Quantization reasoning accuracy GSM8K is not a cliff; it is a tunable dial. Use INT8 when hardware allows, AWQ INT4 when memory forces it, and always evaluate on your own few-shot prompt before shipping. Pair aggressive quantization with self-consistency decoding or external calculators to recover the last points. For most teams, the 2–4x infrastructure savings outweigh a sub-5-point accuracy dip on grade-school math. Ship the quantized model, measure the real failure modes, and keep a full-precision fallback for the queries that actually need it.