Mixture-of-experts models and the size vs speed myth

The assumption that model size linearly dictates latency collapses when you examine sparse architectures. Mixture of experts inference speed is governed by active parameter count per token, not total weights stored, yet the memory footprint tells a more nuanced story that engineers shipping LLM systems must understand before choosing a model.

The dense baseline

A dense transformer applies every parameter to every token. For a 13B parameter model, each forward pass moves roughly 13B weights through the arithmetic units and produces about 2×13B floating-point operations per token (ignoring attention sparsity).

dense_params = 13e9
# Forward FLOPs approx 2 * params per token
dense_flops_per_token = 2 * dense_params

Latency in the decode phase is dominated by memory bandwidth: you must load all 13B weights from HBM to compute a single token. A 70B dense model loads 70B weights, so even if compute is parallelized, the weight-fetch time scales with parameter count. This is why a 70B model feels slower than a 13B model on the same GPU when batch size is one.

How MoE breaks the size-speed link

Mixture-of-experts (MoE) replaces some feed-forward layers with multiple expert networks and a router. Only the top-k experts activate per token. Mixtral 8x7B has ~47B total parameters but activates ~13B per token. The compute cost per token resembles a 13B dense model, while the storage cost resembles a 47B model.

import torch

def route(tokens, router_weight, k=2):
    # tokens: [batch, seq, dim]
    logits = torch.matmul(tokens, router_weight)
    topk = torch.topk(logits, k, dim=-1)
    return topk.indices, torch.softmax(topk.values, dim=-1)

# Mixtral: 8 experts, 2 active
total_params = 47e9
active_params = 13e9
moe_flops_per_token = 2 * active_params  # ~ same as dense 13B

This is the core of the mixture of experts inference speed advantage: you pay for total capacity in VRAM but only pay for subset compute per token.

Where mixture of experts inference speed wins

Under large batch throughput, MoE shines. With many concurrent requests, the GPU’s compute units stay busy because each token picks different experts, and the aggregate active parameters across the batch still fit the compute budget better than a dense model of equal total size.

Consider serving with continuous batching:

python -m vllm.entrypoints.api_server \
  --model mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel

At batch sizes of 32+, Mixtral 8x7B often sustains higher tokens/sec than a dense 34B model on the same hardware, because the 34B model forces 34B active params per token while Mixtral forces ~13B.

Memory bandwidth is the hidden tax

The decode step is memory-bound. To compute one token, you must load the weights of all experts because the router decision happens after the weights are already in cache, or you must load expert weights on demand. Even with expert parallelism, the routing metadata and at least the expert weight pointers reside in memory.

A 47B MoE model requires loading ~47B weights (or shards thereof) into the tensor cores’ vicinity. If your interconnect is slow, the mixture of experts inference speed gain erodes. On a single A100 with 640 GB/s bandwidth, loading 47B fp16 weights takes ~150 ms per layer group; a 13B dense model loads in ~40 ms. The compute saved does not help if you are stalled on weight fetch.

Batching, capacity, and load imbalance

The router is stochastic. Without constraints, one expert can receive 10× the tokens of another, creating a bottleneck. Implementations use a capacity factor:

capacity = int(capacity_factor * batch_size * seq_len / num_experts)
# tokens exceeding capacity are dropped or routed to fallback

If capacity is too low, you drop tokens and lose quality. If too high, you allocate padding and waste memory. In production, we tune capacity_factor between 1.1 and 1.25 for Mixtral-class models.

Load imbalance directly hurts mixture of experts inference speed because the slowest expert dictates the step time for that layer.

Quantization and sharding realities

MoE models quantize unevenly. Experts are often independent, so 4-bit GPTQ on each expert works, but the router is sensitive to precision. We recommend:

• Quantize experts to 4-bit, keep router fp16.
• Shard experts across GPUs with expert-parallel to reduce per-GPU memory footprint.
• Use tensor-parallel only for the shared attention layers.

A typical deployment config:

{
  "model": "mixtral-8x7b",
  "expert_parallel": 8,
  "tensor_parallel": 2,
  "quant": "awq-4bit"
}

Deployment across heterogeneous infrastructure

When you serve dozens of model variants, the gateway layer must understand these internals. An inference gateway such as n4n.ai, which provides a single OpenAI-compatible endpoint across 240+ models, must meter per-token usage accurately despite varying active expert counts, and honors client routing directives to avoid sending latency-sensitive traffic to a degraded MoE provider. That is an operational concern, not a modeling one, but it affects perceived speed.

Tradeoff summary

Decisive takeaway

MoE does not make large models small; it makes them selectively cheap. Mixture of experts inference speed beats dense equivalents only when you can amortize weight loading across enough concurrent tokens and keep experts balanced. For interactive, single-user latency, a dense model of equal active size will be faster because it avoids the routing and memory tax. Choose MoE for throughput-bound serving and quality-per-dollar; choose dense for predictable low-latency edges.