CPU vs GPU inference: why LLMs need parallel compute

The CPU vs GPU inference decision shapes every LLM deployment from prototype to production. GPUs dominate training for good reason — matrix multiplication maps naturally to thousands of cores — but inference economics introduce nuance that benchmarks alone don’t capture. Understanding where each architecture wins lets you right-size infrastructure instead of over-provisioning by default.

Why parallel compute matters for LLMs

Transformer inference is fundamentally a sequence of large matrix multiplications. A 7B parameter model at FP16 requires roughly 14 GB of weights. Each forward pass multiplies hidden states (batch × sequence × hidden_dim) against weight matrices (hidden_dim × 4×hidden_dim for FFN, hidden_dim × hidden_dim for attention). The arithmetic intensity — FLOPs per byte loaded — favors hardware that can keep thousands of multiply-accumulate units fed from high-bandwidth memory.

CPUs compensate with deep caches, sophisticated prefetch, and high single-thread frequency. But a modern Xeon or EPYC core delivers 50–100 GFLOPs FP16 theoretical peak, while an H100 delivers ~2,000 TFLOPs. That 20,000× gap in raw throughput is why CPU vs GPU inference discussions start with parallelism. The gap narrows with quantization and small batch sizes, but the architectural divergence remains.

Latency and throughput profiles

GPU: throughput-oriented, batch-amortized

GPUs excel when you can saturate them. A single A100 80 GB runs Llama-3-70B at FP8 with ~2,000 tokens/sec throughput at batch 32, but first-token latency sits around 50–80 ms. The fixed kernel launch overhead (~5–10 µs) and synchronization barriers mean small batches underutilize the device.

# Typical GPU inference profile (A100 80GB, Llama-3-70B FP8)
batch_size = 1   # ~150 tok/s, 80ms TTFT
batch_size = 8   # ~1,200 tok/s, 120ms TTFT
batch_size = 32  # ~2,000 tok/s, 200ms TTFT

The throughput curve is convex: you pay latency to gain throughput. This makes GPUs ideal for high-concurrency serving where request batching (continuous batching, iteration-level scheduling) keeps the device busy.

CPU: latency-oriented, batch-agnostic

CPUs deliver consistent per-token latency regardless of batch size. An AMD EPYC 9654 (96 cores) runs Llama-3-8B at INT4 around 80–120 tok/s with 30–50 ms time-to-first-token (TTFT) at batch 1. Scale to batch 16 and throughput rises near-linearly to ~1,000 tok/s with minimal TTFT penalty.

# llama.cpp on EPYC 9654, Llama-3-8B Q4_K_M
# Batch 1:  95 tok/s,  38ms TTFT
# Batch 4:  360 tok/s, 42ms TTFT
# Batch 16: 980 tok/s, 55ms TTFT

No kernel launch overhead, no synchronization penalty. This makes CPUs attractive for low-concurrency, latency-sensitive workloads: chat copilots with sporadic traffic, edge devices, or fallback capacity.

Memory bandwidth and capacity constraints

The bandwidth gap (7×) explains why GPU token throughput scales with batch size while CPU throughput is memory-bound at all batch sizes. But the capacity advantage flips for large models: a single CPU node fits Llama-3-405B at INT4 in system RAM; a GPU cluster needs 8× H100s with NVLink.

Quantization narrows the bandwidth gap. INT4/GPTQ/AWQ reduce weight size 4×, making CPU memory bandwidth sufficient for 20–50 tok/s on 70B models. But activation memory (KV cache) still scales with context length and batch size, where GPU HBM retains advantage.

Cost models: cloud and on-prem

Cloud pricing (approximate, us-east-1, on-demand)

Per-million-tokens cost at saturated throughput:

• H100 8×: ~$0.004/M tokens (but requires sustained high concurrency)
• A10G 4×: ~$0.015/M tokens
• CPU r7iz: ~$0.010/M tokens (near-linear scaling, no batching penalty)

The CPU wins on cost per token at low-to-moderate concurrency. The GPU wins when you can sustain >50 concurrent requests and amortize the hardware cost.

On-prem: power and density

An 8× H100 node (DGX H100) draws ~10 kW, needs liquid cooling, rack space, and specialized power distribution. A 2-socket EPYC 9654 server draws ~1.5 kW, fits in standard air-cooled racks. For organizations with existing CPU fleets and no GPU infrastructure, the operational overhead of GPU clusters (drivers, fabric, MIG, DCGM, container runtime) is non-trivial.

Quantization and model support

GPU inference stacks (TensorRT-LLM, vLLM, SGLang) support FP8, INT4 (AWQ/GPTQ), and FP4 (Blackwell) with kernel fusion. Quantization calibration and accuracy validation are mature. But each quantization scheme requires model-specific calibration data and validation — a 70B model quantized to INT4 may lose 2–5% MMLU.

CPU inference (llama.cpp, MLX, ONNX Runtime) supports GGUF quantization (Q4_K_M, Q5_K_M, Q8_0) with one-click conversion. The quantization is post-training, no calibration data needed. Accuracy loss is typically 1–2% for Q4_K_M on 7B–70B models. For teams without ML engineering bandwidth, CPU quantization is lower friction.

# One-command quantization and run on CPU
llama-quantize model.f16.gguf model.q4_k_m.gguf Q4_K_M
llama-server -m model.q4_k_m.gguf -c 8192 -ngl 0

GPU quantization workflow:

# AWQ quantization (requires calibration data, GPU)
python -m autoawq quantize \
  --model_path meta-llama/Llama-3-70B \
  --quant_path ./llama-3-70b-awq \
  --calib_data wikitext2 \
  --w_bit 4 --version GEMM

# Then serve with vLLM
vllm serve ./llama-3-70b-awq --dtype half --max-model-len 8192

Ecosystem and operational ergonomics

GPU stack maturity

vLLM, SGLang, and TensorRT-LLM provide PagedAttention, continuous batching, prefix caching, speculative decoding, and multi-LoRA serving. These features compound: continuous batching + prefix caching + speculative decoding can yield 3–5× throughput over naive implementation. The ecosystem moves fast — new kernels, new quantization formats, new attention backends every quarter.

But operational complexity is high: CUDA version compatibility, driver updates, NCCL/P2P topology, MIG partitioning, DCGM monitoring, Prometheus exporters, Kubernetes device plugins. A GPU serving stack is a distributed systems project.

CPU stack simplicity

llama.cpp, ollama, and vLLM’s CPU backend (experimental) run as single binaries. No drivers, no container runtime, no fabric. Model loading is mmap-based — cold start is disk I/O bound, not kernel compilation bound. Updates are binary replacement.

The tradeoff: no continuous batching (yet), no prefix caching, no speculative decoding, no multi-LoRA. Throughput is memory-bandwidth bound and scales linearly with cores. For internal tools, batch jobs, or low-QPS services, this simplicity often outweighs raw throughput.

When to choose which

Choose GPU when:

High sustained concurrency — >20 concurrent requests, continuous batching amortizes kernel overhead. Chat applications, coding assistants, high-volume API endpoints.

Large models at low latency — Llama-3-405B, Nemotron-3-70B, or MoE models (Mixtral, DeepSeek-V3) where single-stream latency must stay <100 ms TTFT. GPU HBM bandwidth is non-negotiable here.

Speculative decoding and advanced features — Medusa, EAGLE, or draft-model speculative decoding require GPU kernel support. Prefix caching for RAG-heavy workloads. Multi-LoRA for multi-tenant serving.

Training or fine-tuning colocation — If you already run training on GPUs, inference on the same cluster avoids data movement and dual-stack ops.

Choose CPU when:

Low or bursty concurrency — <10 concurrent requests, sporadic traffic, internal tools, batch inference jobs. No batching penalty means you pay only for what you use.

Large models on a budget — Running 70B–400B parameter models at INT4 on commodity servers. A 2-socket EPYC with 1.5 TB RAM costs ~$15k and serves 405B INT4 at 20–30 tok/s. Equivalent GPU capacity costs 5–10×.

Edge, air-gapped, or regulated environments — No GPU drivers, no proprietary blobs, standard x86/ARM hardware. llama.cpp compiles everywhere from Raspberry Pi to SGX enclaves.

Fast iteration and prototyping — Model swap in seconds, no container rebuild, no driver compatibility matrix. GGUF quantization is one command.

Fallback and overflow capacity — n4n.ai routes to CPU-backed workers when GPU pools saturate or degrade, maintaining availability without over-provisioning accelerators.

Hybrid: the pragmatic default

Most production systems end up hybrid. Route high-QPS, latency-sensitive traffic to GPU workers with continuous batching. Route batch jobs, long-context RAG, low-priority traffic, and overflow to CPU workers. Use a gateway that understands model placement, quantization, and per-request routing directives.

# Example routing policy (conceptual)
routes:
  - model: "llama-3-70b"
    quantization: "fp8"
    target: "gpu-pool"
    max_concurrency: 100
  - model: "llama-3-70b"
    quantization: "q4_k_m"
    target: "cpu-pool"
    max_concurrency: 500
    priority: "overflow"
  - model: "llama-3-405b"
    quantization: "int4"
    target: "cpu-pool-large"
    max_concurrency: 20

The CPU vs GPU inference decision isn’t binary — it’s a placement problem. Match the hardware to the workload’s concurrency, latency budget, model size, and operational constraints. The fastest inference is the one that fits your actual traffic pattern, not the benchmark.