Measuring throughput ceiling on 8x H100 clusters

The 8x H100 cluster throughput ceiling is the number every infra team wants before signing the PO, but it is not a spec you read off a datasheet. It emerges from the ugly interaction of tensor parallelism, KV-cache pressure, and request arrival patterns, and you only find the real limit by pushing the node until the NVLink counters stop climbing.

The myth of a single ceiling

Vendors love to quote “max tokens per second” for a box. That number is a lie of omission. A single 8x H100 SXM node has roughly 8 PFLOPs FP16 dense, 640 GB of HBM3, and 900 GB/s NVLink per GPU pair. Those are physics. The throughput ceiling for inference is a function, not a constant: it moves with model size, quantization, batch size, sequence length, and the scheduler’s ability to keep the GPUs fed.

If you treat the 8x H100 cluster throughput ceiling as a static figure, you will either over-provision and waste capex, or under-provision and watch tail latency explode in production.

What actually bounds the 8x H100 cluster throughput ceiling

Compute vs memory bandwidth

For autoregressive decoding, each generated token requires a full read of the model weights (plus attention KV-cache reads). An H100 SXM has 3.35 TB/s of memory bandwidth. A 70B-parameter model in FP16 is ~140 GB. Across 8 GPUs in tensor parallel, each GPU holds ~17.5 GB and reads it per token. The naive memory-bound ceiling per GPU is 3.35 TB/s / 17.5 GB ≈ 191 tokens/s if batch size is 1 and weights are the only thing in flight.

That looks low because it is low for batch=1. Throughput scales when you increase batch size: the weight read is amortized across many concurrent sequences. The compute FLOPs per token become the limiter only at large batches. The real 8x H100 cluster throughput ceiling sits where the scheduler can no longer find independent requests to pack into the same step.

Interconnect saturation

Tensor parallel 8 requires an all-reduce of activations at every layer. NVLink 4.0 gives 900 GB/s bidirectional per connection, but the reduction still consumes cycles. If you launch TP8 without overlapping communication with compute, you leave 10–20% on the table. Profiling with nvidia-smi dmon -s u -o D while watching nvlink rows tells you if GPUs are waiting on each other.

Batch composition and continuous batching

Static batching is dead. Engines like vLLM and TGI use continuous batching: as soon as one sequence finishes, a new one slots in. This changes the ceiling definition. You are no longer measuring “throughput at fixed batch 32”; you are measuring “sustained tokens/sec while the queue is never empty.” That is the only number that maps to a gateway under live traffic.

How to measure it without lying to yourself

Load generation that mirrors production

Synthetic benchmarks that send 512 identical 128-token prompts produce a ceiling that does not exist in reality. Your production traffic has a distribution of input lengths and output lengths. Use a weighted sampler. Drive concurrency high enough that the server’s pending queue is never zero for at least 60 seconds.

Reading the right counters

Don’t trust the client-side tokens/sec alone. Pull GPU utilization, memory bandwidth utilization, and NVLink traffic from DCGM or nvidia-smi. The ceiling is reached when:

• sm_util is >95% on all 8 GPUs
• mem_bw_util is >90%
• NVLink counters stop increasing linearly with load

If any of those are not maxed, you have not found the ceiling; you have found a software bottleneck.

Code: a minimal saturation tester

Below is a load generator that hammers an OpenAI-compatible endpoint with a realistic mix. It uses the official openai python client so it works against vLLM, TGI, or a gateway.

import asyncio, openai, time, random

async def send(client, model, prompt_len, gen_len):
    try:
        await client.chat.completions.create(
            model=model,
            messages=[{"role":"user","content":"x"*prompt_len}],
            max_tokens=gen_len,
            temperature=0,
        )
    except Exception:
        pass

async def main():
    client = openai.AsyncOpenAI(
        base_url="http://8xh100-node:8000/v1", api_key="empty")
    model = "meta-llama/Llama-2-70b-hf"
    tasks = []
    for _ in range(512):
        pl = random.randint(128, 2048)
        gl = random.randint(64, 1024)
        tasks.append(send(client, model, pl, gl))
    t0 = time.time()
    await asyncio.gather(*tasks)
    print("wall time", time.time()-t0)

asyncio.run(main())

Launch the server with tensor parallel explicitly set:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 8 \
  --max-num-seqs 1024 \
  --max-model-len 4096

Using a gateway to pin routing

If you front the node with n4n.ai, you can send a routing directive that pins the benchmark to a specific 8x H100 pool while the gateway still meters per-token usage, so finance gets the cost ceiling alongside the throughput ceiling. That removes client-side retry noise from the measurement.

Tradeoffs: TP8 vs DP8 vs hybrid

TP8 (tensor parallel across all 8 GPUs): required for models that don’t fit in 80 GB at the desired precision. Lowest per-request latency. Highest NVLink dependency. The 8x H100 cluster throughput ceiling under TP8 is sensitive to interconnect health.

DP8 (data parallel, 8 independent replicas): best when the model fits on one GPU (e.g., 13B in FP16, 70B in INT4). Throughput scales near-linearly with replicas. But you lose the ability to serve a single large request fast, and KV-cache capacity is per replica.

Hybrid (PP+TP): pipeline parallel of 2 with TP4 each reduces NVLink pressure but introduces bubble overhead. Only worth it if TP8 all-reduce is the bottleneck, which is rare inside a single node with NVLink.

For a single node, TP8 is the default for 70B+; DP8 is the default for smaller models. Measure both before committing.

What we learned running nightly ceilings

We run a nightly saturation job on a bare-metal 8x H100 node. Three observations hold:

1. Driver and firmware drift moves the ceiling. A CUDA 12.2 to 12.4 jump changed our vLLM throughput by 8% with no code change.
2. Thermal throttling is real. After 20 minutes at full mem bandwidth, one GPU in our sample cluster dropped 3% clocks. The ceiling is a sustained metric, not a 30-second spike.
3. Batch size distribution matters more than average. A workload with 10% long-context (8k) requests caps out at half the tokens/sec of a pure short-context mix, because KV-cache eats HBM and forces smaller batches.

Decisive takeaway

Stop asking “what is the 8x H100 cluster throughput ceiling.” Start asking “what is the ceiling for my batch size distribution, my sequence lengths, and my parallelism config, measured over 30 minutes of saturated load.” Build a saturation harness, pin the model and engine, read the NVLink and memory bandwidth counters, and tune TP vs DP based on model size. The teams that ship reliable LLM infrastructure are the ones who measure the ceiling weekly and leave 15% headroom below it for traffic spikes.