How PagedAttention changed LLM serving throughput

PagedAttention reshaped LLM serving by treating the KV cache like virtual memory pages, and the pagedattention throughput vllm results from early 2023 showed 2–4x higher token rates than contiguous-allocation servers on the same hardware. The thesis is simple: decoupling logical sequence length from physical memory layout is the most important architectural change for high-concurrency inference since kernel fusion. It is not free, and it is not always the right tool, but you should understand the trade before picking a serving stack.

The fragmentation tax in naive KV caches

A transformer generates one KV cache entry per token per layer. In the old model, each sequence got a contiguous slab of GPU memory sized for the maximum context window. A 7B model with 4K context might reserve ~1.5 GB per sequence regardless of how many tokens it actually produced.

That design wastes memory two ways. Internal fragmentation: a 200-token chat reserves the full 4K slot. External fragmentation: after many allocations and frees, the memory pool looks like swiss cheese, and the scheduler rejects new sequences even when free bytes exist.

The practical ceiling on batch size is set by this waste, not by compute. You leave FLOPs on the table because you ran out of KV space.

{
  "seq_1": { "allocated_blocks": 64, "used_tokens": 12 },
  "seq_2": { "allocated_blocks": 64, "used_tokens": 4000 },
  "free_contiguous": 30
}

That fake pool state is typical: seq_1 holds 52 empty blocks, seq_2 is full, and you cannot fit a new 64-block sequence despite 30 free plus 52 reclaimable.

How PagedAttention fixes it

vLLM borrows the page table from operating systems. The KV cache is split into fixed-size blocks (typically 16 tokens). Each sequence maintains a logical-to-physical mapping. Physical blocks need not be contiguous.

# block table: seq_id -> list of physical block ids
block_table = {
    1: [0, 3, 7],
    2: [1, 3],   # block 3 shared with seq 1 (common prefix)
}

Attention kernels gather KV slices via the block table. The GPU pays a small indexing cost, but the memory allocator now packs sequences like a filesystem.

Two wins follow. First, no reservation: a sequence uses exactly ceil(len/block_size) blocks. Second, block sharing: parallel sampling, beam search, or shared system prompts can reference the same physical block for identical prefixes. That sharing directly multiplies effective pagedattention throughput vllm when you run n>1 or many conversations with the same RAG context.

from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-2-7b-chat-hf")
params = SamplingParams(n=4, temperature=0.8)  # 4 completions share prefix KV
outputs = llm.generate("Explain paged attention", params)

The prefix blocks for the prompt are allocated once and referenced by all four outputs.

What the throughput numbers actually mean

The original vLLM paper reported 2.2x–2.4x higher throughput than HuggingFace TGI on ShareGPT traces, and later builds pushed higher. Those are real, but they are not uniform. The gain scales with:

• Request length variance (more variance = more fragmentation saved).
• Batch size (paging lets you pack more sequences).
• Sharing opportunities (beam, parallel samples, common prefixes).

On a fixed batch of equal-length requests at max context, pagedattention throughput vllm converges to the same roofline as any other engine because there is no fragmentation to recover. The benchmark that matters is your production traffic shape, not a synthetic saturated loop.

Tradeoffs you must weigh

Paged attention is not a default win in every regime.

Kernel indirection overhead

Gathering non-contiguous blocks adds pointer chasing in CUDA. For small models (sub-1B) or tiny batch sizes, the launch overhead and irregular memory access can erase the memory win. If you serve a single stream of short requests, a simpler contiguous engine may be faster per request.

Block size tuning

Too small (e.g., 4 tokens) and the block table explodes; too large (e.g., 128) and you reintroduce internal fragmentation. vLLM defaults to 16; for long-context workloads with rare sharing, 32 often helps. This is a knob you must measure, not set once.

Compatibility and ecosystem

Early vLLM lagged on some model architectures and quantization formats. That gap has closed, but if you need a bleeding-edge MoE or custom CUDA graph integration, verify support. TensorRT-LLM and SGLang now implement similar paging, so the idea is portable, but the maturity differs per backend.

Memory accounting under sharing

Shared blocks complicate per-tenant metering. If you bill by token, you must attribute the shared prefix fairly. A gateway that fronts multiple backends—say n4n.ai—can honor client routing directives and forward provider cache-control hints so the shared block is counted once at the origin, not per call.

Deploying it in a real stack

Standing up vLLM is straightforward. The OpenAI-compatible server means existing clients need no changes.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-chat-hf \
  --tensor-parallel-size 2 \
  --block-size 16

Point your OpenAI client at the local port:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[{"role": "user", "content": "Summarize paged attention"}],
    max_tokens=256,
)

For production, put a load balancer in front and scale replicas by KV cache utilization, not just GPU util. Because pagedattention throughput vllm depends on packing efficiency, a replica that is 60% full in memory may already be at its scheduling limit.

When to adopt, when to skip

Adopt paged-attention serving if:

• You serve concurrent users with variable prompt/response lengths.
• You use parallel sampling, beam search, or shared context (RAG, agents).
• Your batch sizes fluctuate and you want GPU cost per token down.

Skip or benchmark carefully if:

• You run a single dedicated stream at fixed length (e.g., offline batch inference on padded inputs).
• Your model is tiny and latency per call dominates.
• You need a feature the paged engine doesn’t yet support.

Takeaway

PagedAttention moved LLM serving from static memory reservation to dynamic paging, and the pagedattention throughput vllm demonstrated is the reason every modern serving framework now copies the design. If you are building inference infrastructure in 2025 and not using a paged KV cache, you are overpaying for GPUs on every variable-length workload. Pick a mature implementation, tune block size to your traffic, and route high-concurrency traffic to it by default.