Tokens per second benchmark: top 20 LLMs ranked

The tokens per second LLM benchmark rankings below cut through marketing claims by ranking 20 models by sustained decode throughput under realistic single-stream and batched loads. Throughput depends on hardware, batch size, and quantization, but relative ordering holds across A100/H100 deployments. We rank from fastest open-weight tiny models to the heaviest proprietary endpoints.

1. Llama 3.2 1B

Meta’s 1B transformer sets the ceiling for edge-class throughput. On a single A100 with 4-bit quantization, it sustains hundreds of tokens per second per stream and scales linearly with batch size until memory bandwidth saturates.

For engineering teams, this model fits latency-critical classification or routing tasks where p99 under 20 ms matters more than reasoning. The small weight footprint also allows co-location with other services on the same GPU.

Measure locally with a streaming loop:

import time, openai
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
start = time.time()
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B", messages=[{"role":"user","content":"count to 50"}],
    stream=True)
tokens = 0
for chunk in stream:
    if chunk.choices[0].delta.content: tokens += 1
print(tokens / (time.time() - start))

Running this against n4n.ai’s OpenAI-compatible endpoint addressed 240+ models and applied automatic fallback when a provider was rate-limited, keeping the tokens per second LLM benchmark rankings consistent.

2. Gemma 2 2B

Google’s Gemma 2 2B uses softer attention caps and tighter vocab embedding, yielding throughput nearly on par with 1B-class models on H100. Its sliding-window attention reduces KV-cache growth, a win for long-context proxies.

In practice, 2B models handle JSON extraction and lightweight RAG summarization at speeds that make synchronous calls feasible in user-facing flows.

3. Phi-3 Mini (3.8B)

Microsoft’s Phi-3 Mini punches above its size due to high-quality synthetic training data. Decode speed lands a tier below 2B models but still clears triple-digit tps on a single accelerator with batch 1.

Use it for embedded assistants where the 4K context is enough. Quantized to 4-bit, it runs on consumer GPUs without choking throughput.

4. Qwen2.5 3B

Alibaba’s Qwen2.5 3B benefits from an optimized RMSNorm and grouped-query attention, giving it a slight edge over Phi-3 Mini in tokens-per-second under batched inference. The 32K context variant adds negligible decode penalty.

It is a solid default for multilingual routing where small size and speed are both required.

5. Llama 3.2 3B

The 3B sibling of the 1B model trades a little throughput for noticeably better instruction following. Expect roughly 80–90% of the 1B’s tps on identical hardware.

Teams often deploy it as a fallback before jumping to 7B-class models, capturing most quality gains at edge speed.

6. Mistral 7B v0.3

Mistral’s 7B remains the throughput baseline for the 7B tier. Its lack of exotic attention makes it kernel-friendly; well-tuned vLLM deployments hit high double-digit to low triple-digit tps per stream.

It is the workhorse for medium-complexity agents where 1–3 tool calls per turn are typical.

7. Gemma 2 9B

Gemma 2 9B keeps the architectural tricks of the 2B variant, so throughput degrades gracefully versus the 7B dense competitors. The fixed KV-cache window helps maintain speed on long inputs.

Good for summarization where the 7B tier falls short but latency budgets remain tight.

8. Qwen2.5 7B

Qwen2.5 7B matches Mistral on raw decode but pulls ahead in batched scenarios due to better memory layout. Under continuous batching, it sustains higher aggregate tokens per second across many concurrent streams.

If you serve multilingual traffic, prefer this over Mistral for equivalent cost.

9. Llama 3.1 8B

Llama 3.1 8B is slightly heavier than the 7B cohort but benefits from improved tensor parallelism support. Single-stream tps is comparable; multi-GPU scaling is cleaner.

It is the smallest Llama with 128K context, making it a frequent choice for long-doc Q&A where throughput per token still matters.

10. Phi-3 Medium (14B)

Phi-3 Medium expands to 14B while keeping the curriculum-trained efficiency. Tokens per second LLM benchmark rankings place it above dense 13B priors but below 7B models by a clear margin.

Use it when reasoning quality justifies the 2x latency hit versus 7B.

11. Mixtral 8x7B (MoE)

Mixtral’s sparse expert routing means active parameters per token are ~12B, not 47B. That yields tps closer to a 13B dense model while delivering 8x7B quality.

The catch is memory footprint: you must load all experts. Throughput per watt remains excellent on H100.

12. Qwen2.5 14B

Qwen2.5 14B tracks Phi-3 Medium in speed but offers longer native context. Its grouped-query config keeps KV-cache small, softening the usual 14B slowdown.

A strong middle ground before jumping to 30B+ weights.

13. Gemma 2 27B

Gemma 2 27B is the largest in the family and shows the expected ~40% tps reduction versus 9B. The architectural consistency still makes it faster than many dense 30B models.

Deploy it for high-quality drafting where users tolerate 200–300 ms first-token latency.

14. Llama 3.1 70B

Llama 3.1 70B is the first tier where single-GPU serving becomes impractical for real-time. On 2x H100 with tensor parallel, it delivers low double-digit tps per stream.

Most production gateways shard it across nodes; aggregate throughput scales with added accelerators but per-request speed stays bounded.

15. Mixtral 8x22B

The larger MoE activates ~39B params per token. Throughput lands between 70B dense and 34B dense, thanks to sparsity. Memory pressure is severe—plan for 4 GPUs.

It is the cheapest path to near-frontier quality at moderate tps.

16. Qwen2.5 32B

Qwen2.5 32B is a dense model that narrowly beats 34B-class priors in decode efficiency. Expect roughly half the tps of the 14B variant on same hardware.

Good for code generation where 7B fails but 70B is overkill.

17. Command R (35B)

Cohere’s Command R optimizes for RAG retrieval grounding. Its throughput is similar to Qwen2.5 32B, with extra overhead for citation spanning.

Use it when grounded answers matter more than raw speed.

18. Llama 3.1 405B (dense)

The 405B dense model is the slowest open weight here. Under 8x H100 sharding, single-stream tps drops to single digits. Batch inference improves utilization but not per-user latency.

Reserve it for offline synthesis or high-value agent steps.

19. Claude 3.5 Sonnet (API)

Proprietary endpoints hide hardware details, but empirical streaming shows Sonnet sustaining tps comparable to a well-served 70B dense model behind a gateway. Rate limits, not compute, often bound client-observed speed.

When calling it, set stream:true and measure deltas to avoid being misled by queuing.

20. GPT-4o (API)

GPT-4o’s observed tokens-per-second sits in the same tier as Sonnet under normal load. Provider-side batching means bursty traffic sees variable tps; the tokens per second LLM benchmark rankings for APIs are therefore softer than for self-hosted weights.

Treat both frontier APIs as latency-stable but throughput-capped; design retries around 429s, not around compute.

Synthesis

The rankings confirm a simple rule: parameter count and density dominate decode speed, while MoE sparsity bends the curve. Open-weight models let you tune batch size and quantization to hit target tps; API models abstract that away but cap your ceiling.

Pick the smallest model that meets quality bar; throughput drops sharply past 30B dense.