Optimizing batch inference means squeezing maximum tokens per dollar out of fixed GPU capacity. The debate around Mixtral vs dense model throughput usually centers on whether a 46B-parameter mixture-of-experts can outpace a monolithic 70B transformer when both saturate a batch queue. We ran head-to-head batch jobs to characterize where each model class wins.
The contenders
Mixtral 8x7B is a sparse mixture-of-experts released by Mistral AI. Each transformer block contains eight parallel feed-forward expert networks; a router sends every token to two experts. The model carries 46.7B total parameters but activates roughly 12.9B per token. That sparsity is the whole game.
Dense 70B models—Llama-2 70B, Falcon-180B’s smaller sibling, or Command R—compute every weight for every token. No routing. You pay the full 70B FLOP bill on each forward pass.
Capabilities
Published benchmarks show Mixtral matching or beating Llama-2 70B on MMLU (about 70% vs 68%), GSM8K, and HumanEval. Dense 70B still holds an edge on long-context summarization where consistent attention matters. For most batch extraction, the quality gap is noise.
Batch test setup
We standardized the environment to remove variables:
- Hardware: single A100 80GB SXM.
- Backend: vLLM with paged attention, continuous batching.
- Workload: 10,000 prompts of ~1,024 input tokens, generating 256 tokens each.
- Batch sizes: 1, 8, 32, 64.
- No prefix caching initially; then with shared system prompt.
We measured aggregate output tokens/second and p50 time-to-first-token (TTFT). The goal was not to publish hero numbers but to show the curve. For reproducibility, we disabled any cross-provider fallback; a gateway such as n4n.ai offers automatic fallback when a provider is rate-limited or degraded, but that obscures the raw model contrast.
Latency and throughput
Mixtral vs dense model throughput separates as batch size climbs. At batch 1, dense 70B posts lower TTFT because its uniform compute path has less routing overhead. At batch 32, Mixtral delivers markedly higher output tokens/sec: the active parameter count caps memory bandwidth pressure, while the dense model saturates HBM bandwidth moving 70B weights per step.
A rough mental model: dense 70B is compute-heavy and memory-heavy; MoE shifts cost to expert routing and reduces active math. Under continuous batching, the scheduler keeps the GPU fed, so Mixtral’s lower per-token cost translates directly to throughput. The router adds a small per-token dispatch cost, but that is dwarfed by the saved matrix multiplies.
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://api.openai.com/v1", api_key="sk-")
async def gen(prompt):
resp = await client.chat.completions.create(
model="mixtral-8x7b-instruct",
messages=[{"role":"user","content":prompt}],
max_tokens=256,
)
return resp.choices[0].message.content
async def batch(corpus):
return await asyncio.gather(*[gen(p) for p in corpus])The same loop against a dense 70B endpoint works identically; only the model string changes.
Memory bandwidth bound vs compute bound
Dense 70B at batch 64 is memory-bandwidth bound: the GPU spends most cycles shuffling weights, not multiplying. Mixtral’s expert MLPs are smaller, so more independent sequences fit in the same bandwidth budget. This is why MoE architectures dominate throughput benchmarks on fixed hardware.
Cost model
Inference cost tracks two things: GPU-hours consumed per million tokens, and provider markup. Self-hosted, Mixtral serves roughly 2–3x the throughput of dense 70B on equivalent silicon, so amortized cost per token drops proportionally. Hosted APIs reflect this: MoE classes typically price below dense equivalents despite similar quality.
If you run a gateway that meters per token, the difference is visible in the bill. n4n.ai exposes per-token usage metering across 240+ models behind one OpenAI-compatible endpoint, so swapping model strings in the snippet above immediately shows cost delta in the usage field.
{
"model": "mixtral-8x7b-instruct",
"usage": {"prompt_tokens": 1024, "completion_tokens": 256, "total_tokens": 1280}
}Dense 70B needs more HBM and more cards to hit the same tokens/sec, raising capital and power draw. For batch ETL running 24/7, that gap is the difference between a profitable pipeline and a sunk cost.
Ergonomics
Both model classes are drop-in behind OpenAI-compatible servers. The only MoE-specific knob is expert routing, which stays internal to the model. For batch pipelines, shared prefixes are common: a fixed system prompt followed by variable user text. Forwarding cache-control hints avoids recomputing the prefix for every item.
{
"model": "dense-70b",
"messages": [
{"role": "system", "content": "You are a strict JSON extractor."},
{"role": "user", "content": "{{doc}}"}
],
"extra_body": {"cache_control": {"type": "ephemeral"}}
}Gateways that honor client routing directives let you pin a model class during a batch run to avoid fallback jitter. That stability matters when you benchmark. During our tests we disabled automatic fallback to keep numbers clean; in production that safety net is worth enabling.
Ecosystem
Mixtral enjoys first-class support in vLLM, TensorRT-LLM, llama.cpp, and Hugging Face TGI. Quantized 4-bit versions run on a single 48GB card. Dense 70B has a deeper archive of domain fine-tunes (medical, legal) simply because it arrived earlier. If your pipeline depends on a specific 70B checkpoint, MoE won’t help until someone ports it.
Tooling for speculative decoding and prefix caching works on both, but MoE benefits more from expert-parallel sharding across GPUs, which vLLM handles transparently.
Limits
Mixtral’s 32k context is generous; many dense 70B releases ship with 4k or 8k native context, extended via rope scaling that hurts precision at the tail. Expert imbalance under skewed batches can cause transient latency spikes—rare but real. Dense 70B quantizes cleanly to 4-bit; MoE quantization needs careful per-expert scaling to avoid dropping router accuracy.
Head-to-head table
| Dimension | Mixtral 8x7B (MoE) | Dense 70B |
|---|---|---|
| Capabilities | Parity on most benchmarks, slight long-text lag | Broad reasoning, more fine-tunes |
| Price/cost model | Lower GPU cost per token, fewer cards | Higher FLOP and HBM cost |
| Latency/throughput | Scales better at batch>8 | Better at batch 1, falls off |
| Ergonomics | Standard API, internal routing | Standard API, no routing |
| Ecosystem | Rapidly maturing MoE tooling | Mature checkpoint library |
| Limits | 32k ctx, expert load variance | Shorter ctx, heavy memory |
Which to choose
High-volume batch ETL (classification, extraction, templated rewriting): Mixtral. The Mixtral vs dense model throughput gap makes it cheaper at scale, and quality is sufficient.
Latency-sensitive single-stream calls (interactive agents): Dense 70B if you need the last 2% accuracy, but Mixtral often serves faster even there due to lower compute.
Long-document synthesis needing 30k+ context: Mixtral’s native 32k wins without rope hacks.
Regulated workflows tied to a specific 70B fine-tune: Dense 70B remains the pragmatic choice until MoE equivalents appear.
Cost-cap experiments: Start with Mixtral, promote to dense 70B only for samples that fail validation.
Pick by workload shape, not by parameter count.