The gap between single-GPU vs multi-GPU throughput is rarely linear, and the right choice depends on model size, batch shape, and tolerance for scheduling complexity. This head-to-head breaks down the tradeoffs with a reproducible test methodology rather than vendor slides, so you can map the numbers to your own serving stack.
Test Methodology
Hardware and Software
We ran tests on a single A100 80GB PCIe card and a 4-GPU A100 80GB NVLink node. Both used CUDA 12.1, NCCL 2.18, and vLLM 0.5.0 with the OpenAI-compatible server front end. The model under test was a 13B parameter dense transformer (fp16), which fits comfortably on one card and shards cleanly across four with tensor parallelism degree 4.
Workload Definition
Throughput was measured as aggregate output tokens per second under fixed concurrency. Each request sent a 512-token prompt and requested 128 generated tokens. Batch sizes (concurrent requests) were 1, 16, and 64. We warmed up the KV cache and discarded the first 10 iterations to avoid cold-start skew.
import asyncio, time, openai
async def send_req(client, prompt, n_tokens):
resp = await client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": prompt}],
max_tokens=n_tokens,
temperature=0
)
return resp.usage.completion_tokens
async def bench(client, concurrency, n_tokens):
prompt = "x" * 2000 # ~512 tokens
start = time.monotonic()
tasks = [send_req(client, prompt, n_tokens) for _ in range(concurrency)]
toks = sum(await asyncio.gather(*tasks))
elapsed = time.monotonic() - start
return toks / elapsed
# asyncio.run(bench(openai.AsyncOpenAI(base_url="..."), 64, 128))Head-to-Head Dimensions
Capabilities
A single GPU caps model size at local VRAM. With 80GB you can serve a 13B model in fp16 or a 70B model in 4-bit, but you cannot run a 180B fp16 model at all. Multi-GPU removes that ceiling via tensor parallelism (TP) or pipeline parallelism (PP). TP splits each layer across GPUs; PP stacks layers. For latency-sensitive batch inference, TP up to 8 GPUs is the pragmatic path on NVLink fabrics.
Price/Cost Model
Cloud hourly rates scale linearly with GPU count: four A100s cost roughly 4x one A100. The nuance is utilization. A single GPU at 30% utilization still burns the full hour; multi-GPU lets you consolidate multiple low-rate single-GPU services onto one node and push aggregate utilization to 70%+. Per-token cost drops only if you actually saturate the extra cards.
Latency/Throughput
Single-GPU vs multi-GPU throughput diverges as batch size grows. At batch 1, one GPU gives the lowest per-request latency because there is no all-reduce overhead. At batch 64, four GPUs typically deliver higher aggregate tokens/sec but slightly higher time-to-first-token due to collective communication. Scaling efficiency for TP on NVLink usually lands in the 70–90% range; on PCIe-only nodes expect 50–70% past two GPUs.
Ergonomics
Single-GPU deployment is a one-line docker run. Multi-GPU introduces NCCL environment variables, GPU affinity, and the risk of a single bad NVLink link stalling the whole job. Orchestrators like Ray or Kubernetes device plugins help, but you now own a distributed system. If you front your inference with a gateway such as n4n.ai, you can issue a routing hint to prefer multi-GPU backends for large batches without changing client code, but the backend complexity remains yours.
Ecosystem
Both setups share the same serving frameworks (vLLM, TGI, TensorRT-LLM). Multi-GPU specifically leans on NCCL, Megatron-LM style TP, and DeepSpeed-Inference. Single-GPU can use lighter runtimes like llama.cpp or ctranslate2. The tooling gap is shrinking, but multi-GPU still demands familiarity with CUDA topology queries:
nvidia-smi topo -mLimits
Single-GPU is limited by memory bandwidth and VRAM. Multi-GPU is limited by interconnect bandwidth and collective op efficiency. Pipeline parallelism adds bubble overhead; tensor parallelism adds all-reduce volume proportional to hidden size. Beyond 8 GPUs on a single node, you hit NUMA and PCIe switch contention unless you move to HGX boards with full NVLink mesh.
Comparison Table
| Dimension | Single-GPU | Multi-GPU |
|---|---|---|
| Capabilities | Fits ≤70B in quant; no TP needed | Runs 100B+ fp16 via TP/PP |
| Price/Cost Model | Linear $/hr, easy to idle | Linear $/hr, better consolidation |
| Latency/Throughput | Best at low batch, lowest TTFT | Higher agg throughput at high batch |
| Ergonomics | docker run simple |
NCCL, affinity, dist-sys ops |
| Ecosystem | llama.cpp, vLLM, TGI | vLLM TP, DeepSpeed, Megatron |
| Limits | VRAM and mem BW ceiling | Interconnect BW, TP overhead |
Reading the Numbers
In our rig, the 13B model on one GPU sustained ~2,400 output tokens/sec at batch 64. The 4-GPU TP4 config sustained ~8,100 tokens/sec at the same batch—a 3.4x scaling factor, not 4x, due to all-reduce. At batch 1, single-GPU TTFT was 18 ms; 4-GPU TTFT was 31 ms. Those are reproducible shapes even if your absolute numbers shift with model and silicon.
The single-GPU vs multi-GPU throughput question is really about batch density. If your traffic is spiky and per-user latency matters, single-GPU keeps the critical path short. If you run nightly bulk extraction across millions of documents, multi-GPU turns a job that takes hours into one that fits a coffee break.
Which to Choose
Prototyping and Low-Traffic Services
Run single-GPU. The operational overhead is near zero, and most 7B–13B models handle p95 traffic under 50 QPS on one card. Use quantization (AWQ or GPTQ) to extend the envelope before reaching for a second card.
High-Volume Batch Pipelines
Choose multi-GPU with tensor parallelism. When you have steady concurrency above 32 and prompts longer than 1k tokens, the aggregate throughput wins justify the NCCL tuning. Prefer NVLink nodes; avoid PCIe-only TP beyond degree 2.
Large Model (70B+) Interactive Apps
Multi-GPU is mandatory. You cannot fit fp16 70B on one 80GB card with a useful KV cache. Use TP8 on a single node or TP4+PP2. Budget for the latency tax on first token and keep batch size modest to hide all-reduce.
Cost-Sensitive Variable Load
Start single-GPU and autoscale horizontally by adding more single-GPU replicas behind a load balancer. This avoids multi-GPU scheduling fragmentation: a 4-GPU node half-empty costs more than two single-GPU nodes full. Only move to single large TP group when batch locality clearly improves utilization.
The decision is not “bigger is better”. Measure your own batch distribution, then match the parallelism degree to the p95 concurrency, not the peak.