How KV cache memory usage scales with context length

The kv cache memory context length relationship is the single biggest determinant of how many concurrent requests your GPU can serve. Every token generated requires reading the entire accumulated key-value history, and that history grows linearly with context. If you’re running inference at scale, you need to know exactly how much VRAM a given context window consumes — and where the breaking points are.

The math is straightforward but unforgiving

For a standard transformer with L layers, H attention heads, and head dimension D, each token adds two tensors per layer: keys and values, each of shape [H, D]. In float16 that’s 2 * L * H * D * 2 bytes per token. For Llama-3-8B (32 layers, 32 heads, 128 head dim), that’s:

2 * 32 * 32 * 128 * 2 bytes = 524,288 bytes per token ≈ 0.5 MB/token

A 128k context window therefore demands ~64 GB of KV cache alone — before you account for model weights, activations, or batch dimension. This is why 8B models on single GPUs hit a hard wall around 32k–64k context in practice.

The formula generalizes cleanly:

def kv_cache_bytes_per_token(num_layers: int, num_heads: int, head_dim: int, dtype_bytes: int = 2) -> int:
    """Bytes per token for standard MHA KV cache (fp16/bf16 = 2 bytes)."""
    return 2 * num_layers * num_heads * head_dim * dtype_bytes

Grouped-query attention (GQA) changes the arithmetic. With G key-value groups (where G < H), the per-token cost becomes 2 * L * G * D * dtype_bytes. Llama-3-70B uses 8 groups across 64 heads, cutting KV memory by 8× relative to MHA. The same 128k context now costs ~8 GB instead of 64 GB — a difference that determines whether the model fits on one H100 or requires tensor parallelism.

Batch dimension multiplies everything

The per-token cost above is per sequence. With batch size B and sequence length S, total KV cache is B * S * bytes_per_token. This is where production systems diverge from benchmark numbers: a benchmark might report “supports 128k context” at batch size 1, but your serving stack runs at batch 32 or 64.

def total_kv_cache_gb(batch_size: int, seq_len: int, bytes_per_token: int) -> float:
    return (batch_size * seq_len * bytes_per_token) / (1024**3)

# Llama-3-8B, fp16, batch 32, 8k context
bytes_per_token = kv_cache_bytes_per_token(32, 32, 128)
print(f"{total_kv_cache_gb(32, 8192, bytes_per_token):.1f} GB")  # ~12.8 GB

At batch 32 and 8k context, the KV cache alone consumes ~13 GB. On an 80 GB H100 with the 8B model weights (~16 GB fp16), you have ~50 GB remaining for activations, workspace, and overhead. That leaves room for perhaps 16k–32k context at this batch size — not the 128k the model architecture theoretically supports.

Prefill vs. decode: different pressure points

During prefill, the entire prompt is processed in one forward pass. The KV cache is written once, but the attention computation is quadratic in sequence length (O(S²)). During decode, each step reads the full cache (O(S)) but only computes attention for one new token.

This asymmetry shapes memory pressure differently:

• Prefill: Peak memory dominated by activation tensors for the S×S attention matrix. For long contexts, activation memory can exceed KV cache memory.
• Decode: Steady-state memory dominated by KV cache. The cache grows by bytes_per_token per step per sequence.

Systems that separate prefill and decode (disaggregated serving) can size each pool independently. A prefill node needs massive activation memory for bursty long prompts; a decode node needs sustained KV cache bandwidth and capacity.

Quantization: the first lever you should pull

FP8 KV cache (E4M3 or E5M2) halves memory with minimal quality loss for most workloads. INT8 and INT4 go further but require calibration and can degrade long-context retrieval. The tradeoff curve:

# Practical fp8 KV cache allocation (pseudocode)
def allocate_kv_cache_fp8(batch_size: int, max_seq_len: int, num_layers: int, num_kv_heads: int, head_dim: int):
    # Shape: [num_layers, 2, batch_size, num_kv_heads, max_seq_len, head_dim]
    # 2 = key + value
    total_elements = num_layers * 2 * batch_size * num_kv_heads * max_seq_len * head_dim
    return torch.empty(total_elements, dtype=torch.float8_e4m3fn, device='cuda')

FP8 is the sweet spot today: 2× capacity increase, native tensor core support on Hopper and Blackwell, and no calibration step. If you’re still running fp16 KV cache on H100s, you’re leaving 2× throughput on the table.

Paged attention: virtual memory for KV cache

vLLM’s paged attention (and derivatives in SGLang, TensorRT-LLM) treats KV cache like OS virtual memory: fixed-size blocks (typically 16 or 32 tokens) allocated on demand. This eliminates fragmentation from variable-length sequences and enables prefix caching — shared blocks for common prompt prefixes.

The memory overhead is the block table: batch_size * max_blocks * 4 bytes (int32 block IDs). For batch 256, 8k context, 16-token blocks: 256 * 512 * 4 = 512 KB — negligible.

# Block allocation logic (simplified)
BLOCK_SIZE = 16
max_blocks = (max_seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE

# Per-sequence block table: [max_blocks] int32
block_tables = torch.full((batch_size, max_blocks), -1, dtype=torch.int32, device='cuda')

def allocate_blocks(num_blocks: int) -> list[int]:
    """Return list of free block indices."""
    free = torch.where(block_pool == -1)[0][:num_blocks]
    block_pool[free] = 1  # mark allocated
    return free.tolist()

Paged attention doesn’t reduce peak memory — it reduces wasted memory. Without it, you allocate for the maximum sequence length in the batch. With it, you allocate only for tokens actually present. At high batch sizes with variable lengths, this recovers 20–40% of KV cache capacity.

Prefix caching: free memory for shared prompts

If multiple requests share a system prompt or few-shot examples, prefix caching reuses the KV blocks for the common prefix. The savings are linear in the shared prefix length:

memory_saved = batch_size * shared_prefix_len * bytes_per_token

For a 2k token system prompt across 64 concurrent requests on Llama-3-8B: 64 * 2048 * 0.5 MB ≈ 64 GB saved. This is why prefix caching is table stakes for any production serving stack — it turns repeated prompt overhead from O(batch) to O(1).

The catch: prefix caching requires exact prefix matches (token-for-token) and increases block table complexity. Most implementations hash the prefix and evict on LRU when the block pool fills.

Sliding window and attention sinks: architectural workarounds

Models with sliding window attention (Mistral, Gemma 2) or attention sinks (StreamingLLM) bound the effective context for attention computation. The KV cache still grows linearly — you still store every token — but the attention pattern only attends to the last W tokens plus a few sink tokens.

This doesn’t reduce KV cache memory. It reduces compute during decode (O(W) instead of O(S)). Don’t confuse the two. If your bottleneck is VRAM capacity, sliding window doesn’t help. If your bottleneck is decode latency at long context, it does.

Context compression: trading quality for capacity

Techniques like KV cache quantization (KIVI, GEAR), token dropping (H2O, SnapKV), or learned compression (LLMLingua) reduce the effective bytes per token. They work by:

1. Quantizing per-channel or per-token with outlier preservation (KIVI: 2.5–4× compression)
2. Dropping low-attention tokens during prefill (H2O: keeps heavy hitters + recent)
3. Compressing prompts before they hit the model (LLMLingua: 2–8× shorter prompts)

These are lossy. The quality degradation is workload-dependent: retrieval-heavy tasks (needle-in-haystack) suffer first; generation-heavy tasks (summarization) tolerate more. In production, treat compression as a tiered fallback: fp8 → int8 → token dropping → prompt compression, each with its own SLA.

The decisive takeaway

KV cache memory scales linearly with context length and batch size, with a coefficient determined by architecture (MHA vs GQA) and precision (fp16 → fp8 → int4). There is no free lunch — 128k context at batch 32 on an 8B model requires ~130 GB of KV cache in fp16, ~65 GB in fp8, ~32 GB in int4.

Your capacity planning should start from this formula:

total_kv_gb = batch_size * max_seq_len * 2 * layers * kv_heads * head_dim * dtype_bytes / 1e9

Then apply the levers in order of cost-effectiveness:

1. FP8 KV cache — 2× capacity, near-zero quality loss, hardware-native
2. Paged attention + prefix caching — recovers 20–40% from fragmentation and shared prefixes
3. GQA-aware model selection — 8× less KV than MHA at same parameter count
4. Compression (quantization → token dropping → prompt compression) — lossy, use as tiered fallback

If you’re building a serving stack, the n4n.ai gateway exposes per-model KV cache coefficients and automatic fp8/int8 quantization so you can size deployments without guesswork. But the formula above is the ground truth — everything else is optimization on top.