Multi-GPU scaling: H100 clusters and inference throughput

Achieving predictable H100 cluster multi-GPU inference throughput means engineering for the system, not the silicon. The common belief that chaining eight H100s yields eightfold token generation ignores the interconnect tax and scheduling contention that dominate at scale.

The bottleneck moves from FLOPS to fabric

An H100 delivers 989 TFLOPS in BF16 and ships with 80 GB of HBM3. Those numbers look great on a datasheet, but a single GPU rarely sustains peak because memory bandwidth, not compute, gates autoregressive decoding. Once you spread a model across multiple GPUs, the limiting factor becomes how fast you can move activations between them.

Inside a node, NVLink 4.0 provides 900 GB/s aggregate bidirectional bandwidth. That is enough to keep tensor parallelism (TP) efficient for models up to ~70B parameters on 8 GPUs. Cross the node boundary and you drop to InfiniBand NDR at 400 Gbps per port—roughly 50 GB/s, or 18x less than NVLink. All-reduce operations that took microseconds now take milliseconds.

# Intra-node TP on a single DGX H100 (8 GPUs)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --tensor-parallel-size 8 \
  --max-num-seqs 256

The moment you issue that command across two nodes with --tensor-parallel-size 16, the same kernel spends a disproportionate slice of each forward pass waiting on fabric.

Tensor parallelism vs pipeline parallelism

Within a node: TP is king

For a single node of 8 H100s, TP splits each layer’s weights and activations across GPUs. The math is simple: each GPU holds 1/8 of the parameters and exchanges intermediate activations after every attention and MLP block. Because NVLink is cheap, communication overlaps well with compute.

Empirically, TP=8 on a 70B model delivers 6–7x the single-GPU token rate for large batches. You lose some headroom to collective overhead, but the scheduling story stays simple: one process group, one model replica.

Across nodes: PP and expert parallelism

Pipeline parallelism (PP) chunks the model vertically—early layers on node A, later layers on node B. It avoids constant all-reduce but introduces bubble time: while node A computes layer 1–10, node B sits idle until the first microbatch arrives. With a small number of microbatches, the bubble can eat 20–30% of wall-clock time.

For Mixture-of-Experts models, expert parallelism spreads experts across GPUs and uses all-to-all communication. This is tolerable on NVLink but painful on InfiniBand unless you heavily overlap with computation.

{
  "strategy": "pipe_parallel",
  "num_nodes": 2,
  "tensor_parallel_size": 8,
  "pipeline_parallel_size": 2,
  "micro_batch_size": 4,
  "num_microbatches": 8
}

That config keeps TP within the node (where NVLink lives) and PP across nodes. It is the least-bad option for 180B+ dense models, but you must tune microbatches to shrink the bubble.

Continuous batching and scheduling overhead

Raw throughput numbers mean nothing if your scheduler leaves GPUs idle between requests. Continuous batching (used by vLLM, TGI, and TensorRT-LLM) packs incoming prompts and generation steps into a rolling batch. The gain over static batching is often 2–4x in real-world traffic.

The tradeoff is CPU cost. The scheduler must iterate over sequences every step, and at 256+ concurrent sequences on a single node, the host process can become the bottleneck. Profile with perf before assuming the GPU is saturated.

# Minimal throughput probe against an OpenAI-compatible server
import asyncio, time, openai

client = openai.AsyncOpenAI(base_url="http://gpu-node:8000/v1", api_key="x")

async def gen(n):
    t0 = time.monotonic()
    tasks = [client.chat.completions.create(
        model="llama-3-70b",
        messages=[{"role":"user","content":"Write a short poem."}],
        max_tokens=128) for _ in range(n)]
    outs = await asyncio.gather(*tasks)
    elapsed = time.monotonic() - t0
    toks = sum(len(o.choices[0].message.content.split()) for o in outs)
    return toks / elapsed

print(asyncio.run(gen(64)))

Run that from a separate machine to avoid contaminating the GPU node’s CPU. If tokens/sec flattens as n grows, you have hit a scheduling or fabric limit, not a compute limit.

Measurement methodology

Never trust a vendor’s “max throughput” sheet. Measure under your own prompt distribution. Three rules:

Use production-like prompt lengths. A 4K-token RAG context stresses KV-cache allocation differently than 32-token chats.
Vary concurrency geometrically: 1, 4, 16, 64, 256.
Watch per-GPU utilization with dcgm-exporter. If nvlink_bandwidth counters climb while sm_util drops, you are fabric-bound.

A well-tuned H100 cluster multi-GPU inference throughput profile shows a knee: throughput rises linearly, plateaus, then degrades as queueing dominates. That knee is your capacity planning anchor.

Tradeoffs: when to scale out vs scale up

Buying a second node is not a default. If your model fits on 8 H100s with TP=8, stay there. Adding nodes multiplies cost and operational complexity for sublinear gain.

Scale out only when:

The model does not fit in 640 GB (8×80) of HBM3 even with quantization.
You need isolation: separate nodes for different tenants to avoid noisy neighbors.
Traffic exceeds what one node’s schedulers can handle on the CPU side.

For 7B–34B models, a single H100 often beats a 4-GPU TP setup on cost efficiency because you avoid collective overhead entirely. Use multi-GPU only when latency at batch=1 is unacceptable and you can feed large batches.

Operational reality: routing and fallback

Cluster throughput is only half the story; request routing decides whether you actually hit those numbers. An inference gateway such as n4n.ai, which offers one OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, still depends on accurate per-cluster throughput estimates to assign load. If the gateway thinks a node can absorb 10k tok/s but the fabric caps it at 6k, requests pile up and fallback fires unnecessarily.

Honor provider cache-control hints and client routing directives. A client that pins a request to a specific TP=8 replica should not be silently shifted to a PP=2 node with different latency characteristics.

{
  "route": {
    "prefer": "tp8-node-a",
    "cache_control": { "read": true, "write": true }
  }
}

That directive, forwarded by the gateway, lets the serving stack reuse KV caches and keeps your measured throughput stable.

Takeaway

Treat H100 cluster multi-GPU inference throughput as a distributed systems problem where the network is the adversary. Start with tensor parallelism inside a single node, use continuous batching, and measure the knee under real traffic. Scale to a second node only when memory or CPU scheduling forces you, and then use pipeline parallelism with carefully tuned microbatches. Anything else is paying 18x interconnect tax for bragging rights.