The H100 SXM vs PCIe inference latency gap is narrower than the price gap suggests for single-GPU calls, but it widens sharply once you split a model across GPUs or sustain high batch loads. Both cards share the same GH100 die and HBM3 bandwidth, so the real differences are power envelope, interconnect topology, and thermal headroom rather than peak FLOPS.
Capabilities: What the Silicon Actually Is
NVIDIA builds one GH100 chip and packages it two ways for the H100 line. The SXM5 module slides into an HGX baseboard; the PCIe card is a standard dual-slot add-in board with a bridge chip for the Gen5 link.
SXM5
- 700 W TDP (configurable to 600 W on some HGX SKUs)
- Four NVLink 4.0 ports, 900 GB/s aggregate bidirectional bandwidth to other SXM GPUs on the same board
- No PCIe root complex on the module; traffic leaves via NVLink or the baseboard’s PCIe switch
PCIe Gen5 x16
- 350 W max TDP (some vendors ship 300 W variants)
- NVLink 4.0 optional via an external bridge, but most PCIe cards ship without it, relying on 128 GB/s PCIe Gen5 to the CPU
- Standard airflow or passive/active cooling, fits commodity servers
The memory subsystem is identical: 80 GB HBM3 at 3.35 TB/s on both. For a single model that fits on one card, the compute ceiling is the same; the SXM simply holds higher clocks longer because it has double the thermal and power budget.
Latency and Throughput
Single-GPU Token Latency
For a 7B–13B model served at batch size 1, the time-to-first-token (TTFT) and inter-token delay are dominated by kernel launch overhead and memory-bound attention. The SXM and PCIe parts post single-digit percentage differences in measured step time because both hit the same HBM bandwidth. The SXM may sustain 10–15% higher boost clock under continuous load, but a well-tuned PCIe card with adequate cooling closes most of that gap.
A minimal latency probe looks like this:
import torch, time
model = torch.load("model.pt").cuda()
inp = torch.randint(0, 32000, (1, 512)).cuda()
torch.cuda.synchronize()
t0 = time.perf_counter()
out = model.generate(inp, max_new_tokens=32)
torch.cuda.synchronize()
print(f"TTFT+decode: {time.perf_counter()-t0:.3f}s")Run that on both form factors under identical software stack; you’ll see the H100 SXM vs PCIe inference latency spread stay under 5% for small batches.
Multi-GPU Tensor Parallelism
This is where the topology bites. Splitting a 70B model across four GPUs with tensor parallel (TP=4) forces an all-reduce per transformer layer. On SXM, NVLink delivers ~1.5 µs hop latency and 900 GB/s bandwidth; on PCIe, you route through the CPU’s PCIe root complex at 128 GB/s with higher software overhead.
The practical effect: a TP=4 decode step on PCIe can be 15–30% slower per token than SXM at high context lengths, because attention activations move across the slower link every layer. If you avoid NVLink on PCIe, the gap is even larger.
Sustained Throughput Under Batch
At batch 64+ the SXM’s 700 W envelope lets it hold base clock through long compute bursts. PCIe cards hit their 350 W limit and clock down, dropping aggregate tokens/sec by 10–20% in power-capped racks. The H100 SXM vs PCIe inference latency difference thus becomes a throughput-per-watt story under load.
Price and Cost Model
SXM does not sell as a loose card. You buy an HGX H100 8-GPU baseboard (or a server with it populated), which bundles the NVSwitch and board. PCIe cards are sold individually and drop into almost any Gen5 server.
- PCIe: lower entry cost, standard warranty, easy to scale one node at a time.
- SXM: higher upfront density cost, but better rack-level performance per socket and per watt.
Cloud hourly rates reflect this: SXM instances command a premium roughly proportional to the sustained throughput gain, not the raw latency gain. For latency-sensitive single-GPU serving, PCIe is the better dollar value.
Ergonomics and Deployment
PCIe wins on ergonomics. You can passthrough the card to a VM, hot-add to a chassis, and cool it with stock fans. SXM demands a specific HGX motherboard, liquid or high-RPM fan wall, and often vendor-locked firmware.
# Check power limit and thermal headroom on either form factor
nvidia-smi -q -d POWER,TEMPERATURE | grep -E "Power Limit|Current Temp"That command reveals the 700 W vs 350 W ceiling immediately. In a colo, PCIe’s lower heat density simplifies rack planning.
Ecosystem and Software Support
Both expose the same CUDA compute capability (9.0), MIG support (with SXM offering full 7 MIG slices per device vs PCIe’s limited or disabled MIG on some SKUs), and NCCL optimizations. NCCL auto-detects NVLink on SXM; on PCIe it falls back to PCIe or optional NVLink bridge. No code changes are required to move a serving stack between them, though you should retune NCCL_P2P_LEVEL and batch size.
When you front a mixed fleet with an inference gateway such as n4n.ai, the node-local microsecond deltas are negligible against client network RTT, but the per-dollar throughput gap still drives routing economics.
Hard Limits and Constraints
- Memory capacity is capped at 80 GB on both; no form-factor advantage.
- SXM’s NVLink is intra-node only; crossing nodes still uses InfiniBand at ~400 Gb/s.
- PCIe cards without NVLink cannot do fast multi-GPU TP; you must accept the PCIe penalty or use pipeline parallel with less frequent sync.
- Thermal: SXM needs 40–55 °C inlet; PCIe tolerates broader ranges but throttles harder if airflow fails.
Head-to-Head Comparison
| Dimension | H100 SXM5 | H100 PCIe Gen5 |
|---|---|---|
| Max TDP | 700 W | 350 W |
| GPU-to-GPU bandwidth | 900 GB/s NVLink (on-board) | 128 GB/s PCIe (NVLink opt.) |
| Single-GPU latency | Baseline (highest boost) | +0–5% typical |
| TP=4 latency penalty | None vs single | +15–30% per token |
| Sustained throughput | Holds clocks | Drops 10–20% when power-capped |
| Entry cost | High (HGX bundle) | Low (single card) |
| Deployment flexibility | Low (vendor chassis) | High (commodity server) |
| MIG slices | 7 full | Often limited/disabled |
| Cooling requirement | Dense, often liquid | Standard airflow |
Which to Choose
Single-model, low-batch inference (≤13B params, batch 1–8): PCIe. The H100 SXM vs PCIe inference latency difference is within noise, and you save capital. Use PCIe cards in a standard Gen5 box and tune for memory bandwidth.
Large-model TP serving (70B+ across 4–8 GPUs): SXM. NVLink’s bandwidth and low latency keep tensor parallel efficient. PCIe’s interconnect tax will erode token throughput enough to offset its cheaper sticker.
Cost-sensitive batch offline jobs: PCIe at scale. If you can pipeline-parallel instead of tensor-parallel, the PCIe link bottleneck shrinks and the lower wattage reduces OpEx.
Latency-critical real-time API with mixed fleet: Either, but standardize on one topology per cluster to avoid NCCL fallback surprises. The microsecond differences rarely matter to the end user; the throughput per watt does.
Edge or colo with limited cooling: PCIe. Half the heat, standard fans, no HGX board dependency.
Pick SXM when interconnect-bound multi-GPU work dominates your bill. Pick PCIe when single-card compute or cost-per-node is the constraint. The silicon is the same; the system around it is not.