The number of experts that activate per token is the single most important knob in a Mixture of Experts architecture. It directly determines the compute-to-parameter ratio, the memory bandwidth pressure during inference, and the model’s ability to specialize. Most production MoE models today activate 2 experts per token, but the latest generation is pushing to 6–8, and the tradeoffs are not what intuition suggests.
What “experts per token” actually means
In a dense transformer, every token passes through every parameter in every layer. In an MoE layer, the feed-forward network (FFN) is replaced by a set of experts — independent FFN blocks — and a router (or gate) that selects which experts process each token. The router outputs a probability distribution over experts, and the top-k experts are chosen. The token’s representation is then computed as a weighted sum of those k expert outputs.
# Simplified MoE forward pass (PyTorch-like pseudocode)
def moe_forward(x, experts, router, k=2):
# x: [batch, seq_len, d_model]
router_logits = router(x) # [batch, seq_len, n_experts]
router_probs = softmax(router_logits, dim=-1)
topk_probs, topk_indices = router_probs.topk(k, dim=-1)
# Normalize top-k weights to sum to 1
topk_probs = topk_probs / topk_probs.sum(dim=-1, keepdim=True)
# Dispatch to experts and combine
out = torch.zeros_like(x)
for i in range(k):
expert_idx = topk_indices[..., i] # [batch, seq_len]
expert_weight = topk_probs[..., i:i+1] # [batch, seq_len, 1]
expert_out = batched_expert_forward(experts, x, expert_idx)
out += expert_weight * expert_out
return outThe critical observation: only k experts execute per token per layer. If a model has 256 experts but k=8, each token sees 8/256 = 3.125% of the FFN parameters per layer. The other 96.875% sit idle for that token. This is where the compute savings come from — but it only materializes if you can actually avoid loading the inactive experts into memory or moving their weights across the memory bus.
The standard: top-2 routing
For years, the de facto standard was k=2. Mixtral 8x7B, Grok-1, and the original Switch Transformer all use top-2 routing with 8 experts per layer. This gives a 4x compute reduction over a dense equivalent (2 active / 8 total) while keeping the router’s job simple: pick the best two.
# Mixtral 8x7B configuration
n_experts = 8
top_k = 2
active_params_per_token = 2 * 7B = 14B # per layer FFN
total_params = 8 * 7B = 56B # per layer FFN
compute_ratio = 2/8 = 0.25 # 4x savingsWhy top-2? Three practical reasons. First, the router only needs to make one meaningful decision — the top choice — and the second choice acts as a hedge. Second, load balancing is tractable: with 8 experts and top-2, each expert expects to receive 25% of tokens (2/8), and the auxiliary loss can keep utilization tight. Third, implementation is straightforward: you can dispatch tokens to two expert kernels and fuse the results without complex scatter-gather patterns.
But top-2 has a ceiling. With only 8 experts, specialization is coarse. Each expert must handle a broad swath of the input distribution, limiting the model’s capacity to develop deep, narrow capabilities. The router also becomes a bottleneck — if it misroutes, half the token’s compute budget is wasted.
The new generation: more experts, higher k
DeepSeek-V2 and V3 changed the calculus. DeepSeek-V2 uses 256 experts per layer with top-6 routing. DeepSeek-V3 pushes to 256 experts with top-8. This shifts the active parameter ratio to 6/256 ≈ 2.3% or 8/256 = 3.125% — dramatically sparser than Mixtral’s 25%.
# DeepSeek-V3 MoE configuration (per layer)
n_experts = 256
top_k = 8
expert_dim = 2048 # each expert is a small FFN
active_params_per_token = 8 * (2 * 4096 * 2048) ≈ 134M
total_params = 256 * (2 * 4096 * 2048) ≈ 4.3B
compute_ratio = 8/256 = 0.03125 # 32x savings vs dense FFNThe motivation is finer-grained specialization. With 256 experts, each can learn a narrow capability — “Python list comprehensions,” “legal contract clauses,” “RNA secondary structure” — rather than broad domains. The router becomes a more sophisticated classifier. But this only works if the router can reliably identify the right 8 experts out of 256 for each token.
DeepSeek achieves this through two mechanisms. First, they use a shared expert — one expert that is always active (effectively k=8+1). This guarantees a baseline of general knowledge and stabilizes training. Second, they invest heavily in router training: a dedicated auxiliary loss that minimizes the coefficient of variation of expert loads, plus a complementary sequence-level load balancing loss that prevents token-level collapse.
# DeepSeek-style load balancing loss (simplified)
def load_balancing_loss(router_probs, topk_indices, n_experts):
# router_probs: [batch, seq_len, n_experts]
# topk_indices: [batch, seq_len, k]
# Token-level: fraction of tokens routed to each expert
tokens_per_expert = torch.zeros(n_experts, device=router_probs.device)
for i in range(topk_indices.shape[-1]):
tokens_per_expert.scatter_add_(0, topk_indices[..., i].flatten(),
torch.ones_like(topk_indices[..., i].flatten()).float())
tokens_per_expert = tokens_per_expert / topk_indices.numel() * topk_indices.shape[-1]
# Probability-level: mean router probability per expert
probs_per_expert = router_probs.mean(dim=(0, 1))
# Coefficient of variation squared * n_experts^2 (Switch Transformer formulation)
loss = n_experts * (tokens_per_expert * probs_per_expert).sum()
return lossRouting mechanisms matter more than k
The number of active experts is only half the story. How they’re selected determines whether the theoretical compute savings materialize.
Top-k with softmax is the baseline. The router outputs logits, softmax produces probabilities, top-k selects. Simple, but the softmax denominator includes all experts, so inactive experts still contribute to the gradient. This is fine for training but means the router “sees” every expert every step.
Sigmoid routing (used in some newer architectures) treats each expert as an independent binary decision. The router outputs n_experts logits, passes through sigmoid, and selects experts above a threshold or top-k. This allows truly sparse gradients — inactive experts receive zero gradient — but makes load balancing harder because there’s no normalization constraint.
Expert choice routing flips the problem: each expert selects its top-k tokens instead of each token selecting experts. This guarantees perfect load balance (every expert gets exactly the same number of tokens) but breaks the standard token-parallel inference pipeline. It’s primarily a training-time technique.
Noisy top-k adds Gaussian noise to router logits during training. This forces the router to develop robust preferences rather than collapsing to a few experts early. At inference, the noise is disabled. This is standard practice in Switch Transformer and derivatives.
Inference implications: where the rubber meets the road
The k value dictates your inference system design. There are three regimes:
Regime 1: All experts fit in GPU memory (small MoE, e.g., Mixtral 8x7B)
With 8 experts × 7B params ≈ 56B FFN params per layer, the full model fits on 2–4 H100s. You load all experts, keep them resident, and dispatch tokens via kernel launches. Latency is dominated by the k expert kernel executions per layer.
# Inference dispatch pattern for resident experts
def moe_inference_resident(x, expert_weights, router, k=2):
# expert_weights: [n_experts, 2, d_model, d_ff] (all in VRAM)
router_logits = router(x)
topk_probs, topk_idx = router_logits.softmax(-1).topk(k, -1)
topk_probs = topk_probs / topk_probs.sum(-1, keepdim=True)
out = torch.zeros_like(x)
for i in range(k):
idx = topk_idx[..., i] # [batch, seq_len]
w = topk_probs[..., i:i+1] # [batch, seq_len, 1]
# Batched expert forward: gather weights, single kernel
expert_w1 = expert_weights[idx, 0] # [batch, seq_len, d_model, d_ff]
expert_w2 = expert_weights[idx, 1] # [batch, seq_len, d_ff, d_model]
hidden = (x @ expert_w1).silu() # [batch, seq_len, d_ff]
expert_out = (hidden @ expert_w2) # [batch, seq_len, d_model]
out += w * expert_out
return outHere, k=2 vs k=8 is purely a compute latency tradeoff. 4x more expert FLOPs per token, but no memory bandwidth penalty for loading weights.
Regime 2: Experts exceed GPU memory but fit in CPU RAM (medium MoE)
DeepSeek-V2/V3 fall here. 256 experts × ~16M params each ≈ 4B FFN params per layer. At FP8, that’s ~4 GB per layer just for FFN weights. With 60+ layers, you’re at 240+ GB — larger than a single node’s VRAM.
You have two options: expert parallelism (shard experts across GPUs, each GPU holds a subset) or expert offloading (keep hot experts on GPU, stream cold experts from CPU/NVMe).
Expert parallelism is the standard approach. With 8 GPUs and 256 experts, each GPU holds 32 experts. For k=8, a token’s 8 experts are distributed across GPUs. This requires all-to-all communication to route tokens to the right GPU, then another all-to-all to gather results.
# Expert parallel all-to-all pattern (conceptual)
def moe_expert_parallel_forward(x, local_experts, router, k=8, ep_group):
# x: [batch, seq_len, d_model] on each GPU
router_logits = router(x) # local router (replicated or sharded)
topk_probs, topk_idx = router_logits.topk(k, -1)
# Convert expert indices to destination GPU ranks
expert_to_rank = topk_idx // experts_per_gpu # [batch, seq_len, k]
# All-to-all: send tokens to GPUs owning their experts
# This is the critical path — latency scales with k and sequence length
dispatched_x, dispatched_probs, dispatched_idx = all_to_all_variable(
x, topk_probs, topk_idx, expert_to_rank, ep_group
)
# Local expert computation
local_out = batched_expert_forward(local_experts, dispatched_x, dispatched_idx)
# All-to-all: send results back to original GPUs
final_out = all_to_all_variable(local_out, dispatched_probs, ..., ep_group)
return final_outThe all-to-all communication volume scales with k × sequence length × hidden size. For k=8, you move 4x more data than k=2. At high batch sizes, this saturates NVLink/NVSwitch bandwidth. This is why DeepSeek uses device-limited routing: they constrain the router to only select experts within the local GPU’s shard for a fraction of tokens, reducing cross-GPU traffic.
Regime 3: Experts exceed CPU RAM (hypothetical future scale)
Not yet relevant for open models, but the trajectory is clear. At 1000+ experts, you need hierarchical routing: a coarse router picks a subset of experts (e.g., 32 of 1024), then a fine router picks k from that subset. This adds router latency but keeps communication bounded.
The specialization vs. utilization tradeoff
Higher k with more experts enables finer specialization, but only if the router can actually learn to use them. The failure mode is expert collapse: the router funnels most tokens to a few “popular” experts, leaving the long tail unused. Those tail experts receive few gradients, never specialize, and become dead parameters.
DeepSeek mitigates this with:
- Shared experts (always active) — absorbs common patterns, reduces pressure on routed experts
- Complementary sequence-level load balancing — forces each expert to be used at least once per sequence
- Higher k — with k=8, even a slightly imbalanced router still touches 8 experts, giving more tail experts gradient signal
But there’s a countervailing force: router capacity. A router that must distinguish 256 experts needs more capacity (larger hidden size, more layers) than one distinguishing 8. DeepSeek uses a separate router network with its own parameters. If the router is too weak, it becomes a random selector, and you lose the specialization benefit entirely.
Memory bandwidth is the real constraint
On modern GPUs (H100, B200), MoE inference is memory-bandwidth bound, not compute bound. The arithmetic intensity of an expert FFN (two matmuls with SiLU) is ~100 FLOPs/byte at FP8. H100 delivers ~3 TB/s memory bandwidth but ~2000 TFLOPs FP8 tensor core. You hit the memory wall long before you saturate compute.
What this means for k: increasing k linearly increases memory bandwidth demand (you must read k experts’ weights), but does not change the arithmetic intensity. If k=2 saturates memory bandwidth, k=8 will be 4x slower with zero compute utilization gain.
The only way k=8 wins is if:
- You use expert parallelism so each GPU reads only its local experts’ weights (1/8 the weight bytes per GPU for k=8 vs k=2 with all experts local)
- You use quantization (FP8/INT4) to reduce weight bytes
- You batch aggressively to amortize weight reads across tokens
In practice, DeepSeek-V3’s k=8 works because they deploy with expert parallelism + FP8 + large batch sizes. For a single-user, low-batch inference scenario, k=8 on a small GPU cluster would be slower than k=2 on the same hardware.
What the major models actually do
| Model | Experts/layer | Top-k | Shared experts | Active FFN params | Routing notes |
|---|---|---|---|---|---|
| Mixtral 8x7B | 8 | 2 | 0 | 14B | Standard top-2, auxiliary loss |
| Grok-1 | 8 | 2 | 0 | ~14B | Similar to Mixtral |
| DeepSeek-V2 | 256 | 6 | 1 | ~134M | Device-limited routing, seq-level balance |
| DeepSeek-V3 | 256 | 8 | 1 | ~134M | FP8, multi-token prediction, MTTP |
| GPT-4 (rumored) | 16 | 2 | ? | ~? | Not publicly confirmed |
The trend is clear: more experts, higher k, shared experts, and sophisticated load balancing. But this only pays off at scale — both model scale (200B+ total params) and deployment scale (expert parallelism across 8+ GPUs).
Decisive takeaway
For most engineers deploying MoE models today: k=2 with 8 experts is the sweet spot. It fits in VRAM on modest hardware, requires no expert parallelism, and the router is trivial to optimize. The 4x compute savings over dense are real and easy to capture.
If you’re training or deploying at DeepSeek scale (200B+ params, 8+ GPU nodes): k=6–8 with 256+ experts and shared experts is the right architecture. The finer specialization improves quality per active parameter, and expert parallelism amortizes the memory bandwidth cost. But you pay for it in system complexity — all-to-all communication, device-limited routing, and a router that itself needs significant capacity.
Do not copy DeepSeek’s k=8 for a 7B–70B model on 1–4 GPUs. You will get slower inference with no quality gain. The router cannot learn meaningful specialization with few experts, and the memory bandwidth penalty will dominate.
The number of experts per token is not a quality knob you turn up freely. It’s a system architecture decision that couples model design, training infrastructure, and deployment topology. Choose the k that matches your GPU count, not your aspirations.