The question of how many parameters fit in 24GB VRAM comes up constantly when engineers size inference workloads. The short answer: at 4-bit quantization you can run roughly 12–13 billion parameters, at 8-bit about 6–7 billion, and at FP16 only 3–4 billion — but those numbers assume zero overhead, which never happens in production. This post breaks down the actual math, the hidden memory consumers, and the architectural decisions that determine whether your model loads or OOMs.
The baseline math
Model weights dominate VRAM usage. Each parameter occupies a fixed number of bytes determined by its precision. The formula is straightforward:
weight_memory_gb = (parameter_count * bytes_per_parameter) / (1024^3)| Precision | Bytes/param | 24GB theoretical max params |
|---|---|---|
| FP32 | 4 | ~6.4B |
| FP16/BF16 | 2 | ~12.9B |
| INT8 | 1 | ~25.8B |
| INT4 | 0.5 | ~51.5B |
These are theoretical ceilings. In practice you lose 15–30% to overhead before loading a single token.
Where the rest of VRAM goes
KV cache
The key-value cache grows with sequence length, batch size, layers, and hidden dimension. For a typical decoder-only transformer:
kv_cache_bytes = 2 * batch_size * seq_len * n_layers * n_kv_heads * head_dim * bytes_per_elementAt FP16 with batch=1, seq_len=4096, 32 layers, 32 KV heads, 128 head_dim: ~1.0 GB. At seq_len=32768: ~8.2 GB. This scales linearly with context — the single biggest variable you control.
Activation memory
Intermediate activations during forward pass consume peak memory proportional to batch × seq_len × hidden_dim. For a 7B model at FP16 with batch=4, seq_len=2048: roughly 1.5–2.5 GB depending on implementation (flash attention reduces this significantly).
Framework overhead
PyTorch’s allocator reserves ~1–2 GB fragmentation buffer. CUDA context: ~300–500 MB. NCCL communicators for multi-GPU: another few hundred MB. These are non-negotiable.
Weight quantization metadata
GGUF/GPTQ/AWQ formats store scale/zero-point tensors alongside quantized weights. For 4-bit group quantization (group_size=128), metadata adds ~0.5–1% to weight size — negligible but non-zero.
Real-world capacity table
Accounting for typical overhead (KV cache at 4K context, activations, framework), here’s what actually loads on a single 24GB GPU:
| Model size | Precision | Fits? | Headroom |
|---|---|---|---|
| 3B | FP16 | Yes | ~14 GB |
| 7B | FP16 | Yes | ~6 GB |
| 13B | FP16 | No | — |
| 7B | INT8 | Yes | ~13 GB |
| 13B | INT8 | Yes | ~5 GB |
| 30B | INT8 | No | — |
| 7B | INT4 | Yes | ~17 GB |
| 13B | INT4 | Yes | ~10 GB |
| 30B | INT4 | Yes | ~2 GB |
| 70B | INT4 | No | — |
The 30B INT4 row is the edge case: it loads with ~2 GB headroom, leaving almost nothing for KV cache beyond ~2K tokens. You’ll hit OOM on the first long request.
Quantization quality tradeoffs
INT4 is not free quality. The perplexity gap versus FP16:
- GPTQ/AWQ 4-bit, group_size=128: ~0.5–1.5% absolute perplexity increase on benchmarks, often imperceptible in chat
- GGML q4_k_m (k-quant mixed): similar, slightly better on outliers
- Naive per-tensor INT4: catastrophic, don’t use
For coding and reasoning tasks, 4-bit 70B outperforms FP16 13B on most evals despite 4× fewer effective bits per parameter. The capacity-quality frontier favors larger quantized models over smaller full-precision ones — up to the point where quantization error accumulates across layers.
Architecture choices that change the math
Grouped-query attention
Models with GQA (Llama-2 70B, Mistral, Gemma) use fewer KV heads than query heads. A 70B GQA model with 8 KV heads needs 4× less KV cache than MHA equivalent. This is why 70B INT4 almost fits on 24GB — the KV cache is manageable at moderate context.
Sliding window / local attention
Mistral’s sliding window caps effective KV cache at window_size × layers. At 4096 window, 32 layers, INT4 7B: ~0.5 GB fixed regardless of actual context length. This changes the capacity equation dramatically for long-context workloads.
Flash attention / paged attention
Flash attention reduces activation peak memory by not materializing the full attention matrix. Paged attention (vLLM, TGI) eliminates KV cache fragmentation, letting you pack 2–3× more sequences in the same VRAM. These are software optimizations that effectively increase your parameter budget.
Multi-GPU changes everything
Two 24GB GPUs (48GB total) with tensor parallelism:
| Model | Precision | Fits? |
|---|---|---|
| 70B | FP16 | Yes |
| 70B | INT8 | Yes |
| 70B | INT4 | Yes |
| 120B | INT4 | Tight |
Tensor parallel splits weights and KV cache across devices. Communication overhead is ~5–10% latency penalty at 7B, ~15–20% at 70B over NVLink. PCIe-only: double that.
Pipeline parallelism splits layers instead. Lower communication but bubble overhead. Rarely used for inference below 8 GPUs.
Decision framework
Start here: What’s your maximum context length and batch size?
available_for_weights = 24GB - kv_cache(max_context, max_batch) - activation_peak - 2GB_frameworkIf available_for_weights < model_weight_size, you have three levers:
- Quantize further (FP16 → INT8 → INT4) — quality cost
- Reduce context/batch — throughput/latency cost
- Add GPU — capital cost
For a chat service targeting 4K context, batch=8, INT4 13B is the sweet spot on 24GB: ~10 GB headroom handles traffic spikes. For code completion with 16K context, drop to 7B INT4 or add a second GPU.
The decisive takeaway
On a single 24GB GPU, the practical parameter ceiling is ~13B at INT4 with 4K context, ~7B at INT4 with 16K context, and ~30B at INT4 only if you accept sub-2K context. FP16 caps at 7B regardless of context. Every additional 4K context costs ~1 GB KV cache at INT4, ~2 GB at FP16.
Size for your maximum context, not your average. The OOM on the 95th-percentile request is what takes down production.