How VRAM requirements scale with parameter count

The VRAM requirements vs parameter count relationship is not a simple multiplication problem. Engineers who treat it as “2 bytes per parameter at fp16” end up OOM at runtime because they forgot the KV cache, activation memory, and framework overhead. Understanding the full stack — weights, quantization, context-dependent KV cache, and runtime allocations — is the difference between a model that fits on your 24 GB card and one that doesn’t.

The baseline math starts at 2 bytes per parameter

At full fp16 or bf16 precision, each parameter occupies 2 bytes. A 7B model needs 14 GB for weights alone. A 70B model needs 140 GB. This is the floor before any quantization, before the KV cache, before a single token of context.

# Raw weight memory at fp16/bf16
def weight_memory_gb(param_count_billions: float) -> float:
    return param_count_billions * 2  # 2 bytes per parameter

# 7B  -> 14 GB
# 13B -> 26 GB
# 34B -> 68 GB
# 70B -> 140 GB

This math assumes dense parameters. Mixture-of-experts models like Mixtral 8x7B report 47B total parameters but only 13B active per forward pass. The weight memory for active parameters is ~26 GB at fp16, but you still need all 47B parameters loaded (or streamed) — so the VRAM requirement sits closer to 94 GB unless you offload experts to CPU or NVMe.

Quantization changes the weight equation entirely

Quantization is the single biggest lever on VRAM requirements vs parameter count. The industry has converged on 4-bit as the practical default for local inference, with 8-bit for quality-sensitive workloads and 3-bit/2-bit for extreme compression.

GGUF formats add metadata and non-quantized tensors (embeddings, output norm, lm_head often stay fp16), so actual usage runs 10–15% above the naive bit-packed calculation. A “4-bit” 7B GGUF typically consumes 4.2–4.5 GB in practice.

# Real ggml tensor layout for a 7B q4_k_m model
# tensor          shape           type        size
# token_embd      32000 x 4096    f16         256 MB
# output_norm     4096            f16         32 KB
# output          32000 x 4096    f16         256 MB
# blk.0.attn_qkv  4096 x 12288    q4_k        192 MB
# blk.0.attn_out  4096 x 4096     q4_k        64 MB
# blk.0.ffn_gate  4096 x 11008    q4_k        172 MB
# blk.0.ffn_down  11008 x 4096    q4_k        172 MB
# blk.0.ffn_up    4096 x 11008    q4_k        172 MB
# ... (32 layers)
# Total: ~4.3 GB on disk, ~4.5 GB in VRAM with context

The quality tradeoff is real but often overstated. At 4-bit, most models retain 95–99% of fp16 benchmark scores. At 3-bit, degradation becomes noticeable on reasoning and coding tasks. At 2-bit, coherence collapses for anything beyond chat. Choose quantization based on your quality floor, not just VRAM pressure.

KV cache scales with context, not parameter count

This is where engineers get surprised. The KV cache grows with context length, batch size, and number of layers — not parameter count directly. For a transformer with L layers, H heads, d head dimension, context length T, batch size B, at precision p bytes:

KV cache = 2 * L * B * T * H * d * p
         = 2 * B * T * (hidden_size) * p * L

Since hidden_size * L correlates with parameter count, KV cache does scale with model size, but the multiplier is context length. At fp16 (p=2):

Batch size multiplies this linearly. A batch of 4 at 32K context on a 70B model needs 80 GB just for KV cache at fp16. This is why vLLM, TGI, and SGLang implement PagedAttention and KV cache quantization (FP8, int8, or int4 KV) — without them, long-context inference is impossible on consumer GPUs.

# KV cache estimation for planning
def kv_cache_gb(layers: int, hidden: int, context: int, batch: int = 1, bytes_per_elem: int = 2) -> float:
    # 2 * layers * batch * context * hidden * bytes_per_elem
    return (2 * layers * batch * context * hidden * bytes_per_elem) / (1024**3)

# Llama-3-70B: 80 layers, 8192 hidden
# 32K context, batch=1, fp16 -> 20.0 GB
# 32K context, batch=1, fp8  -> 10.0 GB
# 32K context, batch=4, fp8  -> 40.0 GB

KV cache quantization to FP8 (1 byte per element) halves this with minimal quality loss. int4 KV is experimental but shows promise for extreme context. If you’re serving long contexts, budget 50–100% of weight memory for KV cache at fp16, or 25–50% at FP8.

Activation memory and framework overhead add 10–30%

Activations during forward pass scale with batch size, sequence length, and hidden size. For training, activations dominate memory. For inference, they’re smaller but non-trivial — especially with large batch sizes or long prefill chunks.

Framework overhead is the silent killer. PyTorch’s allocator reserves a memory pool (default 80% of free VRAM). CUDA context takes 300–500 MB. The tokenizer, embedding tables, and logits buffer add overhead. vLLM’s block manager reserves contiguous blocks. TGI’s sharded tensors add replication factor.

# Rough inference overhead budget
def inference_overhead_gb(model_size_gb: float, context_len: int, batch: int) -> float:
    base = 0.5  # CUDA context, tokenizer, buffers
    allocator_reserve = 0.1 * model_size_gb  # PyTorch pool fragmentation
    activation_estimate = (batch * context_len * 8192 * 2) / (1024**3)  # rough
    return base + allocator_reserve + activation_estimate

A safe rule: add 15–20% on top of weights + KV cache for fp16 inference. For quantized models where weights are small, the relative overhead grows — a 4-bit 7B model at 4.5 GB weights might need 1.5 GB overhead, a 33% tax.

Real-world GPU mapping: what actually fits

Theoretical calculations meet hardware reality at the VRAM boundary. Here’s what fits on common GPUs with 4-bit quantization and 4K–8K context, batch=1, FP8 KV cache:

Multi-GPU changes the math via tensor parallelism (split weights across GPUs, each holds 1/N weights + full KV cache) or pipeline parallelism (split layers, each holds full layer weights + partial KV). Tensor parallel is standard for inference; pipeline adds latency.

# vLLM tensor parallel example: 70B q4 on 2x 24 GB
# Each GPU: ~17.5 GB weights + ~2.5 GB KV (4K ctx, FP8) + overhead = ~22 GB
# Fits with headroom

# 70B fp16 on 4x 80 GB (A100)
# Each GPU: ~35 GB weights + ~5 GB KV (4K ctx, fp16) + overhead = ~42 GB
# Fits comfortably

Apple Silicon unified memory changes the game: a 96 GB or 192 GB Mac Studio runs 70B q4 or 70B q8 entirely in “VRAM” with no GPU-to-CPU transfer penalty. The bandwidth (400–800 GB/s) is lower than H100 (3 TB/s) but sufficient for single-user inference.

The decisive takeaway

VRAM requirements vs parameter count is a three-term equation: quantized weights + KV cache(context, batch, precision) + overhead. Weights are the only term proportional to parameter count. KV cache scales with context and batch. Overhead scales with framework choices.

For capacity planning:

1. Start with quantized weight size — 0.5× params for int4, 1× for int8, 2× for fp16.
2. Add KV cache — 2 × layers × batch × context × hidden × kv_precision_bytes. Use FP8 KV as default.
3. Add 15–20% overhead — more for small models, less for large.
4. Round up to the next GPU tier — fragmentation and allocator behavior make “it fits on paper” unreliable at >90% utilization.

A 70B model at int4 with 32K context, batch=1, FP8 KV: ~35 GB weights + ~10 GB KV + ~7 GB overhead = 52 GB. That needs 2× 24 GB (tensor parallel) or 1× 80 GB. The same model at 4K context: ~35 + ~1.25 + ~5.5 = 42 GB — fits on a single 48 GB card.

Don’t guess. Calculate the three terms, pick your quantization and context policy, then buy the GPU that leaves 10–15% headroom. Anything less is debugging OOM errors at 2 AM.