A Mixtral 8x7B speed benchmark rarely tells the story engineers actually need. Raw tokens-per-second numbers splashed across provider landing pages ignore tail latency, concurrency collapse, and quantization artifacts that surface only in production.
The thesis: speed is more than tokens per second
If you are shipping a feature backed by Mixtral 8x7B, you care about predictable latency under load, not a best-case number from a cold start. The model is a sparse mixture-of-experts (MoE) architecture: 8 experts per layer, 2 active per token. That design cuts compute per token compared to a dense 47B model, but it places unusual pressure on memory bandwidth and scheduler fairness.
A meaningful Mixtral 8x7B speed benchmark must measure at least three things:
- Time to first token (TTFT), which exposes queueing and prefill cost.
- Inter-token latency (ITL) at your real batch size.
- Stability of both as concurrency climbs.
Providers optimize different points. A provider tuned for single-stream low latency will lose to a batch-optimized one on aggregate throughput, and vice versa. Picking a backend from a single-number leaderboard is how teams end up with a demo that works and a production outage.
What Mixtral 8x7B actually costs to serve
MoE inference quirks
Mixtral 8x7B has ~46.7B total parameters but only ~12.9B active per forward pass. The weights still must live in GPU memory. At fp16 that is ~93 GB, so a single 80 GB A100/H100 cannot hold it unquantized with much room for KV cache. Most providers quantize to fp8 or int4, or shard across two GPUs.
That has direct speed consequences. fp8 on H100 keeps most of the math in tensor cores; int4 with GPTQ/AWQ saves memory but adds dequant overhead and can drop quality on tricky routing. The speed benchmark you run on a fp8 endpoint will not match an int4 endpoint’s token rate, nor its output distribution.
KV cache and context
Mixtral’s 32k context window means KV cache grows fast. A provider that allocates a large static cache per request will hit memory limits earlier, forcing preemption. You will see ITL spikes that never appear in a 128-token test. If your application sends long system prompts, measure with those prompts, not a toy string.
How to measure properly
Don’t trust a single curl with a stopwatch. Write a client that opens N concurrent streams, streams responses, and records timestamps per token.
Code for a minimal benchmark
import asyncio, time, openai
async def stream_one(client, prompt, model):
start = time.monotonic()
first_token = None
tokens = 0
async with client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=256,
stream=True,
) as resp:
async for chunk in resp:
if chunk.choices[0].delta.content:
if first_token is None:
first_token = time.monotonic()
tokens += 1
end = time.monotonic()
return {
"ttft": first_token - start,
"itl_avg": (end - first_token) / max(tokens - 1, 1),
"tokens": tokens,
}
async def run_bench(model, concurrency, base_url, api_key):
client = openai.AsyncOpenAI(base_url=base_url, api_key=api_key)
prompts = ["Explain Rust ownership in 200 words."] * concurrency
results = await asyncio.gather(*[stream_one(client, p, model) for p in prompts])
return results
# results = asyncio.run(run_bench("mistralai/mixtral-8x7b-instruct", 8, "https://provider-a.example/v1", "sk-..."))This gives you per-request TTFT and average inter-token latency. The max_tokens=256 caps generation so the test finishes; adjust to your real output length.
Metrics that matter
Aggregate with percentiles, not averages. A single slow request behind a lock can drag the mean while 99% of users are fine.
{
"provider": "A",
"concurrency": 8,
"ttft_p50_ms": 120,
"ttft_p99_ms": 900,
"itl_p50_ms": 18,
"itl_p99_ms": 120,
"errors": 0
}Replace the numbers with your observed values. The p99 ITL is where user-facing streaming dies. If p99 ITL is 5x p50, your users will see stutter even if the median feels snappy.
Statistical rigor
Run each concurrency level at least 30 times. Use a warm-up phase of 5 requests to let autoscalers spin up. Plot TTFT and ITL against concurrency on a log-log scale; the knee of the curve is your real capacity. A provider that stays flat to 16 concurrent but explodes at 24 is different from one that degrades linearly from 4.
Provider landscape: what differs under the hood
Quantization and kernels
One provider may serve Mixtral with TensorRT-LLM fp8 on H100, another with vLLM + AWQ on A100. The former will show lower TTFT at high batch because the kernel fusion is aggressive. The latter is cheaper to run, so the provider may offer lower price but throttle concurrency.
A Mixtral 8x7B speed benchmark that does not record the served precision is useless. Ask the provider or infer from output logprobs variance. If you see repeated token loops on edge cases, suspect aggressive int4.
Batching and concurrency
Continuous batching in vLLM or TGI changes the equation: your request shares the GPU with others. At low traffic you get near-dedicated speed. At peak you get scheduled behind larger batches. Run your benchmark at 3 PM and 3 AM; the curves differ because the multi-tenant GPU is fuller during business hours.
Region and network
An endpoint in us-east-1 adds ~70 ms to a eu-west client before a token is generated. Measure from the same region as your production caller. Use a warm-up request to avoid cold container spins. If your client is serverless, its own cold start can dwarf model TTFT.
Cache and prefix reuse
Some providers support prefix caching for system prompts. If your benchmark sends the same 500-token system prompt every call, a cached prefill turns TTFT from hundreds of ms to tens. A Mixtral 8x7B speed benchmark without cache hints measures the wrong thing if your real traffic reuses prefixes. Forward cache-control where the API allows; gateways can propagate those hints to the backend.
Tradeoffs: raw speed vs reliability
Fast endpoints often cap concurrency per key. You get 200 tokens/s on one stream, but the second stream gets 429. A slower endpoint with elastic scaling may sustain 80 tokens/s across 50 streams without error.
If your app is a single-user copilot, optimize for TTFT and ITL p50. If you run bulk summarization, optimize for total throughput per dollar and tolerate higher latency. Quantization tradeoffs are real. int4 Mixtral can be 2x faster on memory-bound hardware, but we have seen expert routing degrade on code tasks. Run a task-specific eval, not just a speed benchmark.
Using a gateway to smooth over variance
Running the same Mixtral 8x7B speed benchmark against five providers means five client configs, five auth schemes, and five failure modes. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and automatically falls back when a provider is rate-limited or degraded, while honoring your routing directives and forwarding cache-control hints. That lets you benchmark by flipping a header instead of rewriting code.
Even without a gateway, log per-provider routing so you can attribute latency spikes to the right backend.
Decisive takeaway
Stop comparing headline tokens-per-second. Stand up a concurrent streaming benchmark from your production region, capture TTFT and ITL at p50 and p99 under your real concurrency, and record the served precision. Then choose the provider that holds p99 ITL flat as load rises, not the one with the best single-stream number. If you need to hedge across backends, put a routing layer in front so a degraded provider doesn’t take down your latency budget. Mixtral 8x7B is fast enough everywhere; the differentiator is consistency under pressure.