What is GPU inference and how does it work

GPU inference is the process of running a trained neural network on a graphics processing unit to generate predictions, using the GPU’s massive parallelism to compute matrix multiplications across thousands of cores simultaneously. Unlike training, which updates weights through backpropagation, inference only performs the forward pass — making it a throughput- and latency-sensitive workload rather than a memory-bandwidth-bound one. Understanding this distinction shapes every hardware and software decision downstream.

How GPU inference works

At its core, inference is a sequence of matrix multiplications and pointwise operations. A transformer layer, for example, computes attention scores (QKᵀ), applies softmax, multiplies by values (AV), then passes through feed-forward networks — all reducible to batched GEMM (general matrix multiply) calls. GPUs excel here because each output element is independent, mapping naturally to SIMT (single instruction, multiple threads) execution.

The hardware pipeline looks like this:

1. Host prepares inputs — token IDs, attention masks, position IDs — and copies them to device memory (VRAM) via PCIe or NVLink.
2. Kernel launch — the runtime (CUDA, ROCm, or a graph compiler like TensorRT/XLA) enqueues a sequence of kernels: GEMM for linear layers, custom kernels for fused attention, elementwise ops for activations and normalization.
3. Execution on SMs — each streaming multiprocessor (SM) runs warps of 32 threads in lockstep. Tensor cores (on Volta and later) accelerate mixed-precision GEMM by accumulating FP32 while multiplying FP16/BF16/INT8.
4. Memory hierarchy — weights stay in VRAM; activations flow through L2 cache, shared memory, and registers. The bottleneck is almost always memory bandwidth, not compute.
5. Output copy — logits or embeddings return to host, or stay on device for the next iteration in autoregressive decoding.

A minimal PyTorch inference path illustrates the flow:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",  # handles multi-GPU sharding
).eval()

prompt = "Explain GPU inference in one sentence."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

device_map="auto" uses Accelerate to shard layers across available GPUs — critical when the model exceeds single-GPU VRAM. torch.inference_mode() disables autograd, saving memory and enabling kernel fusions.

Why GPU inference matters for LLM serving

Three constraints define production LLM serving: latency (time to first token), throughput (tokens/second across concurrent requests), and cost per million tokens. GPUs dominate all three for models above ~1B parameters because:

• Tensor cores deliver 10–100× the FP16/BF16 throughput of CPU AVX-512 units.
• High-bandwidth memory (HBM2e/3/3E at 1.5–5 TB/s) feeds the compute units; CPUs top out around 200–400 GB/s with DDR5.
• Batching — GPUs amortize kernel launch overhead and weight loading across many requests. Continuous batching (iterative scheduling) keeps SMs saturated while handling variable-length sequences.

The economics are stark: an H100 (80 GB) serves ~2,000–4,000 tokens/s on Llama-3.1-8B at batch sizes that saturate the device. A dual-socket Xeon server might manage 200–400 tokens/s for the same model at 3–5× the power draw. For any real traffic, GPU inference is the only viable option.

Concrete example: serving Llama-3.1-70B at scale

Consider a team deploying Llama-3.1-70B (BF16, ~140 GB weights) behind an OpenAI-compatible API. The model doesn’t fit on one GPU. Options:

A typical vLLM deployment with TP=4 on H100s:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384

--enable-chunked-prefill splits long prefill chunks across iterations, preventing OOM on large contexts while keeping decode throughput high. --max-num-batched-tokens controls the continuous batching window.

At n4n.ai, we see teams underestimate the KV cache footprint: 2 bytes/token/layer/head (BF16) × 80 layers × 64 heads × 8K context ≈ 80 GB for a single 70B request. Multi-GPU tensor parallel splits this, but the aggregate VRAM pressure remains. Planning for peak concurrency × context length is where capacity models break.

Common misconceptions

“Inference is just matrix multiplication — any GPU works”

Consumer GPUs (RTX 3090/4090) have tensor cores but lack NVLink, ECC, and multi-GPU scaling. For 70B+ models, you need 80 GB VRAM per GPU and high-bandwidth interconnect (NVLink/NVSwitch) for tensor parallelism. PCIe 4.0/5.0 adds 5–15 μs latency per all-reduce step — acceptable for batch inference, fatal for low-latency serving.

“Quantization is free accuracy”

INT4/FP8 quantization reduces VRAM and increases throughput, but quality drops non-linearly. For Llama-3.1-70B, AWQ INT4 loses ~2–4 points on MMLU vs BF16; FP8 (with per-tensor scaling) loses ~0.5–1. The tradeoff is real — test on your eval set, not just benchmarks.

“Bigger batch = better throughput always”

Throughput scales with batch size until you hit:

• SM occupancy limits (register pressure, shared memory)
• KV cache capacity (VRAM)
• Scheduler overhead (kernel launch, Python GIL in naive loops)

vLLM and TensorRT-LLM use continuous batching to keep the device saturated without monolithic batches. The sweet spot for H100 on 70B is often 128–256 concurrent sequences, not thousands.

“CPU offload solves VRAM limits”

Offloading layers to CPU RAM (via device_map="auto" with offload_folder) works for batch inference with no latency SLA. For serving, the PCIe round-trip per layer adds 10–50 ms/token — unusable for interactive workloads. Use it for eval, not production.

“FlashAttention is a drop-in replacement”

FlashAttention-2/3 fuses QKᵀV into one kernel, reducing HBM reads/writes by 2–4×. But it requires:

• Causal mask (no arbitrary attention patterns)
• Head dimension ≤ 128 (FlashAttn-3 supports 256)
• No per-head bias or ALiBi without kernel mods

If your model uses grouped-query attention (GQA) with non-standard head dims, you may fall back to the slower xFormers or PyTorch SDPA path. Check the kernel compatibility matrix.

The software stack matters as much as hardware

Raw GPU specs don’t determine served performance. The stack — runtime → compiler → scheduler → kernel library — decides whether you hit 80% or 30% of peak FLOPS.

A practical rule: start with vLLM or SGLang. They integrate continuous batching, FlashAttention, CUDA Graphs, and speculative decoding out of the box. Only drop to TensorRT-LLM or custom Triton kernels when you’ve profiled and identified a specific kernel as the bottleneck.

Capacity planning checklist

Before ordering GPUs, answer:

1. Model size (quantized) — weights + KV cache per request at max context.
2. Target concurrency — simultaneous requests at p99 latency budget.
3. Parallelism strategy — TP degree, PP stages, expert parallelism (for MoE).
4. Interconnect — NVLink domain size (8 for H100, 4 for A100); cross-node needs InfiniBand/RoCE.
5. Power/cooling — H100 SXM is 700W; 8-GPU node needs 10–12 kW + overhead.
6. Software SLA — does your stack support the quantization, context length, and scheduling features you need?

Skip any of these and you’ll be debugging OOMs or latency spikes in production.

GPU inference is not “run the model on a GPU.” It’s a systems problem spanning memory hierarchy, interconnect topology, kernel fusion, scheduling policy, and quantization tradeoffs. The hardware provides the ceiling; the software stack determines where you actually land. Teams that treat inference as a first-class engineering discipline — profiling, capacity planning, and iterating on the serving stack — ship lower latency at lower cost. Everyone else over-provisions and under-delivers.