MoE vs dense models: what's the tradeoff?

The tradeoff between MoE vs dense models comes down to a single architectural decision: whether to activate all parameters for every token or route each token through a subset of specialized experts. That choice cascades into training compute, inference latency, memory footprint, and the operational complexity of serving the model in production. Understanding where each architecture pays off — and where it creates new problems — is essential for any team choosing a foundation model or designing a serving stack.

What makes MoE different from dense models

A dense transformer activates every parameter for every forward pass. A 70B dense model performs 70B parameter operations per token, full stop. MoE replaces the monolithic feed-forward network in each layer with multiple expert networks — typically 8 to 256 of them — and a lightweight router that selects a small subset (usually 1 or 2) per token. The total parameter count can be massive (Mixtral 8x7B has 47B total parameters), but only a fraction (13B active) participates in any given inference step.

This sparsity is the lever. It lets you scale total model capacity without linearly scaling compute. But it introduces routing overhead, load-balancing constraints, and a fundamentally different memory access pattern that changes everything about how you serve the model.

Parameter count vs active parameters

The most common confusion in MoE vs dense models discussions is conflating total parameters with active parameters. A dense model’s parameter count equals its active parameter count. An MoE model’s total parameters can be 4-8x higher than its active count.

# Dense: 70B params = 70B active per token
# MoE (Mixtral 8x7B): 47B total params = 13B active per token (2 of 8 experts)
# MoE (DeepSeek-V2): 236B total params = 21B active per token (6 of 160 experts)

This distinction matters for two reasons. First, model quality correlates more strongly with active parameters than total parameters — a 13B-active MoE competes with a 13B-dense model, not a 47B-dense model. Second, serving infrastructure must hold the total parameters in memory (or stream them from disk), even though compute only touches the active subset. You pay the memory cost of the full model but the compute cost of the active portion.

Training and inference economics

Training an MoE model is cheaper per token for a given total parameter count because you only backpropagate through active experts. The router gradients are negligible. DeepSeek-V3’s training run demonstrated this: 2.788M H800 GPU-hours for a 671B-total-parameter model (37B active), compared to estimated 30M+ GPU-hours for a hypothetical 671B dense equivalent.

Inference economics flip the script. For dense models, cost scales linearly with parameter count. For MoE, cost scales with active parameters — but only if your batch size is large enough to amortize the memory bandwidth cost of loading all expert weights. At small batch sizes (typical for interactive chat), you’re bandwidth-bound loading experts that may not even be used for the current token. The router adds a small but non-zero latency penalty: a matrix multiply and softmax per layer per token.

# Rough inference cost comparison (illustrative, not benchmarked)
# Dense 70B: 70B params * 2 bytes (bf16) = 140 GB VRAM, 70B FLOPs/token
# MoE 8x7B: 47B params * 2 bytes = 94 GB VRAM, 13B FLOPs/token active
# But MoE needs all 94 GB resident; dense only needs 140 GB for 70B

Latency and throughput characteristics

MoE shines at high throughput, struggles at low latency. The router creates a data-dependent memory access pattern: each token in a batch may activate different experts, scattering memory reads across the full expert weight matrix. This kills cache locality and makes small-batch inference slower than an equivalent-active-parameter dense model.

At large batch sizes, the math changes. You can fuse expert computation across tokens that happen to route to the same expert, recovering utilization. vLLM and SGLang implement expert parallelism and batch scheduling specifically for this. But if your workload is single-stream chat with batch size 1-4, a dense model often delivers lower tail latency.

# Pseudocode: MoE routing creates irregular memory access
def moe_forward(x, experts, router):
    # x: [batch, seq, hidden]
    router_logits = router(x)                    # [batch, seq, num_experts]
    topk_idx, topk_weights = topk(router_logits) # [batch, seq, k]
    
    # Scatter-gather: each token pulls different expert weights
    # This is the latency killer at small batch sizes
    expert_outputs = []
    for i, expert in enumerate(experts):
        mask = (topk_idx == i).any(dim=-1)       # Which tokens use this expert?
        if mask.any():
            expert_out = expert(x[mask])         # Irregular batch sizes per expert
            expert_outputs.append((mask, expert_out, topk_weights[mask, i]))
    
    return combine(expert_outputs)

Memory and hardware requirements

MoE models demand more VRAM per active parameter than dense models because you must store all experts simultaneously. An 8x7B MoE needs ~94 GB VRAM (bf16) for 13B active parameters — ~7.2 GB per active billion. A 13B dense model needs ~26 GB — ~2 GB per active billion. The overhead comes from inactive experts occupying memory without contributing compute.

This has practical consequences for GPU selection. A dense 70B model fits on 2×H100 (80GB each) with room for KV cache. An 8x7B MoE fits on 2×H100 but leaves less headroom for context. Larger MoEs (DeepSeek-V2 236B total, 21B active) need 4-8×H100 just for weights, before KV cache. Expert parallelism across GPUs adds NVLink/PCIe traffic for the all-to-all communication that routes tokens to the correct expert shard.

Quantization helps both architectures, but MoE benefits more from per-expert quantization since experts see different token distributions. AWQ and GPTQ can target 4-bit or 3-bit with minimal quality loss, cutting the 94 GB to ~24-32 GB for Mixtral-class models.

Model quality and capabilities

The quality question is settled: at equal active parameter counts, MoE matches or slightly exceeds dense models on benchmarks. Mixtral 8x7B (13B active) beats Llama-2 13B and competes with Llama-2 34B. DeepSeek-V2 (21B active) beats Llama-3 70B on code and math. The specialization effect is real — experts learn distinct capabilities (coding, reasoning, languages) and the router learns to dispatch appropriately.

But MoE has failure modes dense models don’t. Router collapse — where the router ignores most experts — wastes capacity. Load imbalance — where a few experts get overwhelmed — creates hot spots. Training stability is trickier; the auxiliary loss that encourages balanced routing can conflict with task loss. DeepSeek-V2 and Mixtral both use sophisticated load-balancing losses (expert-level and device-level) that dense models simply don’t need.

For long-context tasks, MoE’s sparse activation helps: fewer active parameters means less KV cache per token for the same total model capacity. But the router must handle position-dependent routing correctly, which adds complexity to attention implementations.

Ecosystem and tooling maturity

Dense models win on ecosystem maturity. Every inference engine (vLLM, TGI, TensorRT-LLM, llama.cpp) supports dense transformers natively. MoE support is newer and uneven:

• vLLM: Added MoE support in 0.4.x, expert parallelism in 0.5.x. Production-ready for Mixtral-class models.
• TGI: Supports MoE via text-generation-inference router, but expert parallelism requires custom sharding.
• TensorRT-LLM: Strong MoE kernels, but configuration is verbose.
• llama.cpp: CPU/Metal MoE support works, but GPU offload of experts is all-or-nothing per layer.
• SGLang: Purpose-built for MoE with radix attention and expert scheduling; fastest for high-throughput MoE serving.

Fine-tuning tooling is similarly split. LoRA on MoE requires deciding whether to adapt the router, all experts, or a subset. PEFT libraries support this but best practices are still emerging. Dense model fine-tuning is a solved problem with thousands of documented recipes.

Comparison table

Which to choose by use case

Choose dense when:

• Serving interactive chat with batch sizes ≤ 8 and strict latency SLAs (p50 < 100ms, p99 < 500ms)
• Running on limited VRAM (single GPU, consumer hardware, or 2×A100/H100)
• Fine-tuning frequently with LoRA/QLoRA and needing battle-tested recipes
• Building a prototype where operational simplicity outweighs marginal quality gains
• Your model size target is ≤ 34B parameters (dense models in this range are highly optimized)

Choose MoE when:

• Running high-throughput batch workloads (code generation, embedding, async completion) where batch size ≥ 32
• You need > 70B active parameter quality but have VRAM for only 20-40B active equivalent
• Serving multiple specialized tasks where expert specialization pays off (code + multilingual + reasoning)
• You can invest in expert-parallel serving infrastructure (SGLang, vLLM with EP) and tune capacity factors
• Training from scratch or continued pre-training where compute budget favors sparse scaling

The hybrid reality: Most production systems will serve both. A dense 8B-13B model handles low-latency chat and edge deployment. An MoE 8x7B or larger handles high-throughput background tasks and quality-critical completions. Route requests by latency budget and quality requirement, not by architecture religion.