Qwen 3 32B throughput benchmark across providers

Running a Qwen 3 32B throughput benchmark across providers is less about finding a single winner and more about mapping your workload shape to the right deployment topology. The 32B parameter class sits in a sweet spot—capable enough for agentic loops, small enough to fit on a single high-VRAM node with quantization—but throughput varies by an order of magnitude depending on batching, concurrency, and provider scheduling.

Why throughput beats latency for 32B class models

Engineers often fixate on time-to-first-token (TTFT) from a single curl call. That metric is nearly useless for capacity planning. A provider can cache the prompt and front-load a fast first chunk while starving subsequent tokens under load.

Throughput—tokens per second per request and aggregate tokens per second across the fleet—determines whether your inference bill scales linearly or explodes. For a 32B model, the decode phase dominates cost because each token requires a full forward pass over the weights. Unlike a 7B model, the larger weight matrix saturates memory bandwidth on most accelerators, making decode a memory-bound rather than compute-bound problem.

A Qwen 3 32B throughput benchmark that only reports median TTFT will mislead you into choosing a provider that falls over at eight concurrent users.

Benchmark methodology

You need a reproducible harness before comparing numbers. Below is a minimal async Python client that hits an OpenAI-compatible endpoint, streams completions, and measures wall-clock tokens/sec under concurrency.

import asyncio, time, openai

async def stream_one(client, prompt, model):
    start = time.monotonic()
    chunks = 0
    async for chunk in await client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    ):
        if chunk.choices[0].delta.content:
            chunks += 1
    return chunks / (time.monotonic() - start)

async def run_bench(model, base_url, key, conc, prompts):
    client = openai.AsyncOpenAI(base_url=base_url, api_key=key)
    tasks = [stream_one(client, p, model) for p in prompts[:conc]]
    return await asyncio.gather(*tasks)

# usage: asyncio.run(run_bench("qwen3-32b", "https://api.provider.com/v1", "sk-...", 16, ["Explain RPC."]*32))

This measures per-request decode rate under a fixed concurrency level. Run it against each provider with identical prompts and context lengths. Capture the mean and p95 across at least three runs.

Controlling for context and batch size

Qwen 3 32B, like its predecessors, exhibits quadratic attention cost in prefill. A 4K context prompt prefills slower than a 512-token one. Fix the input size across runs. Also disable provider-side prompt caching unless you are explicitly testing cache hit paths—caching masks the raw compute cost.

{
  "model": "qwen3-32b",
  "messages": [{"role": "user", "content": "<fixed 1024-token lorem>"}],
  "stream": true,
  "cache_control": {"type": "ephemeral", "ttl": 0}
}

If your gateway honors client routing directives, pin the provider for the test so fallback doesn’t skew the Qwen 3 32B throughput benchmark. A routing header might look like:

{ "routing": { "provider": "fixed", "target": "vendor-a" } }

Provider landscape: what actually moves the needle

Managed API clusters

Managed providers run optimized serving stacks—vLLM, TensorRT-LLM, or proprietary schedulers. They amortize KV cache across requests via continuous batching. The upside: high aggregate throughput without ops work. The downside: you inherit their queueing latency and rate limits.

A Qwen 3 32B throughput benchmark on these endpoints typically shows flat per-request throughput until saturation, then rapid collapse. That knee is the metric to record. Note whether the provider exposes batch APIs; some let you submit 100 prompts in one request, which can double effective throughput by avoiding per-request overhead.

Self-hosted single GPU

A single 80GB GPU with 4-bit quantization serves the model with low overhead. You control batch size. Without a tuned scheduler, you leave 30–50% utilization on the table because the GPU waits on memory-bound decode. Still, for predictable low-concurrency internal tools, this is cheapest per token at steady state.

Multi-GPU tensor parallel

Spreading the 32B weights across two or four GPUs reduces per-device memory pressure and improves prefill parallelism. Throughput per watt peaks here, but inter-GPU all-reduce becomes the bottleneck for small batches. Use this only if you have sustained concurrency above 8–16 streams.

Quantization and memory bandwidth

The decode step for a 32B model moves roughly 32B parameters × bytes-per-param from VRAM to compute per token. At 4-bit, that is ~16GB per token; at FP16, ~64GB. H100 bandwidth is 3.35TB/s, so theoretical max decode is ~200 tokens/sec at 4-bit, but overhead and attention KV reads cut that sharply. The Qwen 3 32B throughput benchmark you run will land well below theoretical—expect memory-bound behavior to dominate.

Concurrency and continuous batching

The defining feature of any production-serving stack is continuous batching. Traditional static batching waits for a full batch before decoding; continuous batching inserts new requests as soon as the scheduler finds a free slot. This is why a managed endpoint can sustain 20 concurrent streams at near-single-stream per-request speed, while a naive self-hosted script degrades linearly.

When you run your own Qwen 3 32B throughput benchmark, sweep concurrency from 1 to 64. Plot per-request tokens/sec vs concurrency. A healthy stack stays above 70% of single-stream rate until the saturation point. If you see immediate degradation from 1 to 2 streams, the scheduler is broken or the instance is oversubscribed.

Tradeoffs: cost, control, and degradation

Self-hosting gives you fixed cost and data locality but demands monitoring of GPU health, model updates, and request shaping. Managed providers charge per token and may throttle during peak. Serverless GPU offerings abstract the node but add cold-start penalties that wreck TTFT for sporadic traffic.

Degradation behavior matters more than median throughput. A provider that sheds load by dropping connections forces you to implement retry-with-backoff; one that queues requests silently inflates tail latency. Your benchmark should include a fault-injection pass: kill a fraction of requests mid-stream and measure recovery.

A brief comparison:

Using a gateway for resilience

If you standardize on an OpenAI-compatible endpoint, an inference gateway can route around degraded providers. n4n.ai, for example, provides automatic fallback when a provider is rate-limited and forwards cache-control hints, so a Qwen 3 32B throughput benchmark on the gateway layer reflects blended capacity rather than a single vendor. That is useful for production, but isolate single-provider numbers first to know what you are actually buying.

Decisive takeaway

Run a Qwen 3 32B throughput benchmark with realistic concurrency before signing up for any provider. If your traffic is sparse and interactive, prioritize TTFT and pick a managed endpoint with edge caching. If you run batch extraction or high-parallel agent swarms, self-host with continuous batching or use a managed stack that publishes its scheduler type. Never trust a vendor’s marketed tokens/sec—measure the knee, measure the tail, and own the harness.