n4nAI
Open console →
Benchmarks & performanceAnalysisQwen Speed and Throughput Benchmarks

Qwen 3 235B benchmark: speed and throughput

A practitioner's analysis of Qwen 3 235B benchmark performance: how MoE architecture affects latency and throughput, with real serving tradeoffs.

n4n Team5 min read1,162 words

Audio narration

Coming soon — every post will get a voice note here.

Any serious evaluation of Qwen 3 235B benchmark performance has to separate single-request latency from saturated-throughput numbers. The model’s mixture-of-experts design changes both axes in ways that trip up naive benchmarks, and most published numbers conflate the two. If you are sizing GPUs or setting SLOs, the distinction is the difference between a cost-effective deployment and a silent money pit.

The architecture dictates the curves

Qwen 3 235B is a sparse Mixture-of-Experts model (specifically, Qwen3-235B-A22B). It carries 235 billion parameters in total but activates only ~22 billion per token. That single fact explains most of its benchmark behavior.

Active vs total parameters

A dense 22B model and the active path of Qwen 3 235B do similar compute per forward pass. The difference is that the 235B weights must live in GPU memory and be fetched per expert group. Compute cost per token is low; memory footprint is enormous.

This means prefill (processing the prompt) is bounded by how fast you can load expert weights and route tokens, not by raw FLOPS. Decode (generating one token at a time) is memory-bandwidth bound because you still must read the full parameter set for the activated experts plus the shared components.

Memory bandwidth bound decode

On an H100 with ~3.3 TB/s bandwidth, reading 22B active params in FP8 (~22 GB) takes ~7 ms purely in weight fetch, before any math. Add the attention KV cache reads and you get floor latency around 10–15 ms per token at batch size 1 on a single shard. That is the baseline any Qwen 3 235B benchmark performance claim must acknowledge.

Quantization changes the memory math

The total parameter count determines static memory. In FP16, 235B weights need ~470 GB across GPUs. In FP8 that drops to ~235 GB; in INT4 (via AWQ or GPTQ) it is ~118 GB. The active expert weights scale similarly: 22B active is ~44 GB FP16, ~22 GB FP8, ~11 GB INT4.

Running FP8 is the pragmatic default on H100/H200. It halves the memory pressure and lets you fit the model on 8 GPUs with headroom for KV cache. INT4 squeezes it onto 4 GPUs but typically costs a few points of accuracy on reasoning tasks—a tradeoff you must validate per workload.

Benchmark methodology that doesn’t lie

Most misleading numbers come from measuring average tokens/sec across a mixed batch without isolating phases. You need two separate measurements.

Prefill vs decode separation

Prefill throughput should be measured in tokens processed per second on long prompts with zero generation. Decode throughput should be measured as generated tokens per second under fixed concurrency. Report both, plus the cross-over point where they interfere.

When reporting Qwen 3 235B benchmark performance, always state the concurrency, context length, and quantization. A number without those is advertising, not engineering.

Code for measuring token throughput

Below is a minimal async probe that saturates a deployment with concurrent requests and computes aggregate decode throughput. It uses the standard OpenAI client against any compatible endpoint.

import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://your-endpoint/v1", api_key="sk-...")

async def gen(req_id: int):
    t0 = time.monotonic()
    resp = await client.chat.completions.create(
        model="qwen3-235b-a22b",
        messages=[{"role": "user", "content": "Explain the Carnot cycle in detail."}],
        max_tokens=512,
        temperature=0.0,
    )
    t1 = time.monotonic()
    return resp.usage.completion_tokens, t1 - t0

async def main(concurrency: int):
    tasks = [gen(i) for i in range(concurrency)]
    results = await asyncio.gather(*tasks)
    total_tok = sum(r[0] for r in results)
    # use max wall time as conservative batch completion
    wall = max(r[1] for r in results)
    print(f"Concurrency {concurrency}: {total_tok/wall:.1f} tok/s aggregate")

asyncio.run(main(32))

Run this at concurrency 1, 8, 32, 128. Plot the curve. If throughput climbs linearly until it plateaus, you’ve found the expert-parallel saturation point.

Real-world throughput tradeoffs

Serving this model is a game of placing experts across GPUs to minimize all-reduce traffic.

Batch size and expert parallelism

With 8-way tensor parallelism and expert parallelism, a single node of 8x H100 can hold the model in FP8 with leftover memory for KV cache. Continuous batching (as in vLLM or SGLang) hides the expert routing latency by overlapping compute across requests.

The win: at high concurrency, Qwen 3 235B benchmark performance often shows markedly higher total token throughput than a dense 70B model on the same hardware, because the active param count is lower and FLOPS are underutilized anyway. The cost: tail latency at low QPS is worse than a small dense model because of expert invocation overhead and larger weight footprints per shard.

Latency at low QPS

If your traffic is spiky and you care about p99 under 200 ms for short responses, a 235B MoE is the wrong tool. A 22B dense or 32B model will give tighter latency at fraction of the GPU cost. The MoE only pays off when you have steady concurrent streams that amortize the fixed memory reads.

Expert imbalance is a real risk

MoE routers are not perfectly balanced. In production traces we have seen one expert group receive 3x the traffic of another, causing localized compute spikes. Inference engines mitigate this with expert parallelism and token dropping, but you should monitor per-expert load. If your benchmark uses a single homogeneous prompt, it will hide this entirely.

Tuning the serving stack

A baseline vLLM launch for a single 8-GPU node looks like:

vllm serve Qwen/Qwen3-235B-A22B \
  --tensor-parallel-size 8 \
  --expert-parallel-size 8 \
  --quantization fp8 \
  --max-num-seqs 256 \
  --max-model-len 32768

The --max-num-seqs value is your concurrency ceiling. Push it too high and KV cache allocation fails; too low and you leave GPUs idle. On 8x H100 with FP8, 256 sequences at 32k context is aggressive but feasible if average generated length stays under 1k.

SGLang often edges out vLLM on MoE routing efficiency due to its built-in expert parallelism scheduler. Benchmark both against your own traffic before committing.

Serving across providers without rewriting apps

In production you rarely control a single cluster. You may have Qwen 3 235B deployed on one vendor’s H100 pool and another’s A100 fallback. If you front these with an OpenAI-compatible gateway that honors client routing directives and forwards provider cache-control hints—such as n4n.ai—you can shift traffic during provider degradation without changing a line of application code. The benchmark numbers you collected per endpoint stay comparable because the API surface is identical.

That said, fallback is not free. When a primary provider fails mid-stream, you either abort or implement resume with prompt caching. Forwarding cache-control hints matters: Qwen’s long prefill is expensive, and a gateway that drops cache_control breaks your cost model.

Comparison to other large MoEs

Mixtral 8x22B has 141B total parameters and ~39B active. Qwen3-235B-A22B has a lower active ratio (22/235 vs 39/141), which is why its per-token compute is smaller despite a larger footprint. The trade is that Qwen needs more memory bandwidth to feed experts, making it more sensitive to interconnect topology. On NVLink-connected nodes it thrives; on PCIe-only links, expert all-to-all traffic stalls.

Honest tradeoff summary

  • Pros: High aggregate throughput under load; strong quality at 22B active params; cheaper per-token compute than dense equivalents of similar quality.
  • Cons: High static memory cost; worse single-request latency; complex expert placement; needs high concurrency to shine.
  • Operational risk: MoE routing bugs in inference engines still appear; vLLM versions matter. Pin your stack.

Decisive takeaway

Qwen 3 235B benchmark performance is excellent when you measure it as a throughput machine under concurrent load, and mediocre when you measure it as a low-latency single-stream responder. Deploy it behind a batcher, keep concurrency above ~16, and use a routing gateway if you span providers. If your workload is mostly isolated requests with tight latency SLAs, pick a smaller dense model and save the GPUs.

Tagsqwen-3throughputbenchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceQwen 3 32B throughput benchmark across providersA practical analysis of Qwen 3 32B throughput benchmark across providers, covering methodology, concurrency, and how to pick the right deployment.Benchmarks & performanceQwen 3 benchmark performance for coding workloadsAnalysis of Qwen 3 benchmark performance coding workloads: throughput, latency, and real-world tradeoffs for engineers shipping LLM apps.Benchmarks & performanceQwen 3 benchmark performance on long-context promptsAnalysis of Qwen 3 benchmark performance long context: accuracy holds up, but throughput and latency need prefix caching and batching to stay viable.Benchmarks & performanceQwen 3 benchmark performance under concurrent loadAnalyze Qwen 3 benchmark performance concurrent load: how dense and MoE variants scale under parallelism, where latency breaks, and how to serve them.Benchmarks & performanceQwen 3 benchmark performance vs Qwen 2.5 comparedEngineering comparison of Qwen 3 vs Qwen 2.5 benchmark performance across capabilities, cost, latency, and ergonomics with a verdict for production LLM deployments.Benchmarks & performanceQwen 3 vs Llama 4 Scout: throughput comparedHead-to-head comparison of Qwen 3 and Llama 4 Scout on throughput, cost, latency, and ergonomics, with a verdict for engineering teams.

All qwen speed and throughput benchmarks posts →