Llama 3.1 405B tokens per second across major providers

Llama 3.1 405B tokens per second is the metric every team quotes when comparing inference providers, but the headline numbers rarely survive contact with production traffic. The gap between a provider’s marketing benchmark and your actual throughput stems from quantization choices, batching dynamics, and context length, not from dishonesty so much as from mismatched workloads.

The hardware floor for 405B

Llama 3.1 405B in fp16 weighs about 810 GB (405B params × 2 bytes). You cannot fit that on one accelerator, so every serving stack uses tensor parallelism across at least 8 H100/H200 GPUs. Decode is memory-bandwidth bound: each generated token requires reading the full weight set. An H100 delivers ~3.35 TB/s of memory bandwidth. Split across 8 GPUs, each holds ~100 GB and must read it per token step, giving a theoretical single-stream ceiling around 30–40 tokens per second before accounting for attention KV cache, communication overhead, and software stack inefficiencies.

That math explains why no provider delivers hundreds of tokens per second to a single isolated request. Aggregate cluster throughput is a different number entirely.

What “tokens per second” actually measures

Most published Llama 3.1 405B tokens per second figures are aggregate: total completion tokens served across a saturated batch divided by wall-clock time. That number looks great on a slide and terrible for your latency-sensitive agent.

You need to separate three quantities:

• Prefill throughput – tokens per second while processing your prompt. This is compute-heavy and scales with batch size.
• Decode throughput (per stream) – the speed at which one user’s completion grows. This is the memory-bound number above.
• Time to first token (TTFT) – dominated by prefill, not decode.

A provider can post a massive aggregate decode number by packing 64 concurrent requests, while your single-user stream crawls at 20 tps. Always ask which curve you are looking at.

Major providers, qualitatively

I’m not going to invent benchmark tables; the numbers shift weekly as teams tune kernels. But the architectural differences are stable:

• Together AI and Fireworks run heavily optimized fp8/int8 paths on H100 clusters, trading minor accuracy for higher batch efficiency. Their Llama 3.1 405B tokens per second under load is typically better than fp16-only hosts.
• AWS Bedrock and Azure AI Studio expose Meta’s reference fp16 deployment. You get bit-exact weights and enterprise compliance, but lower density per GPU and consequently higher cost per token.
• Groq and similar LPU vendors serve smaller Llama variants; 405B is often unavailable or constrained because their wafer-scale design favors lower-parameter models. Check availability before architecting.
• Self-host on H200 or MI300X shifts the burden to you but removes per-token margin.

The takeaway: the same model weights produce wildly different throughput depending on the serving stack.

Quantization and its speed tradeoffs

fp8 cuts memory footprint and bandwidth roughly in half versus fp16, directly improving both prefill and decode density. int4 goes further but introduces noticeable quality regression on reasoning tasks—measurable on GSM8K or HumanEval, not just vibes.

If you serve RAG answers where the model mostly formats retrieved text, int4 might be fine. If you run autonomous code agents, fp8 is the pragmatic floor. The speed delta shows up precisely in the Llama 3.1 405B tokens per second curve: at equal concurrency, fp8 yields ~1.8× the batch throughput of fp16 on the same hardware.

# Rough cost/quality decision matrix (illustrative)
configs = {
    "fp16": {"weight_gb": 810, "rel_speed": 1.0, "quality_risk": "none"},
    "fp8":  {"weight_gb": 405, "rel_speed": 1.8, "quality_risk": "low"},
    "int4": {"weight_gb": 203, "rel_speed": 3.1, "quality_risk": "medium"},
}

Measure it like an engineer

Stop reading provider blogs. Write a harness that replays your real prompts. Use the OpenAI-compatible API surface most hosts now provide.

from openai import OpenAI
import time, statistics

client = OpenAI(base_url="https://api.together.xyz/v1", api_key="KEY")

def measure(model, prompt, n=5):
    tps = []
    for _ in range(n):
        start = time.time()
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=512,
        )
        elapsed = time.time() - start
        ct = resp.usage.completion_tokens
        tps.append(ct / elapsed)
    return statistics.median(tps)

print(measure("meta-llama/Llama-3.1-405B-Instruct-Turbo", "Explain Raft."))

Run that against each provider with your own prompt distribution and concurrency. The resulting Llama 3.1 405B tokens per second will reflect your TTFT tolerance, your average context length, and your batch size—not a synthetic 32-token reply.

Streaming reveals more. Capture delta arrival times to see decode stability under load:

stream = client.chat.completions.create(model=model, messages=..., stream=True)
first = None
last = None
count = 0
for chunk in stream:
    if chunk.choices[0].delta.content:
        now = time.time()
        if first is None: first = now
        last = now
        count += 1
decode_tps = (count - 1) / (last - first)  # excludes TTFT

Routing and resilience in practice

When you benchmark three providers and pick one, you still lose when that provider has a bad hour. An inference gateway such as n4n.ai exposes a single OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is rate-limited or degraded, letting you script failover without rewriting clients. That matters more than a 10% tps advantage that disappears under incident.

Honor provider cache-control hints and set routing directives explicitly. If your workload is latency-critical, pin to fp8 hosts; if it’s cost-critical, allow int4 fallback.

Tradeoffs you actually care about

• Latency vs cost: Higher tps often means more aggressive batching, which raises your TTFT. Measure p50 and p99, not averages.
• Compliance vs flexibility: Bedrock/Azure keep data in your tenant; startups may not.
• Reproducibility: fp16 reference weights give identical outputs across runs; quantized paths may vary by kernel.

The decisive factor is workload shape. A 4k-token prompt with 200-token answer stresses prefill; a 200-token prompt with 4k-token answer stresses decode. The same provider can look best or worst on each.

Takeaway

Benchmark your own traffic, not the provider’s. The only honest Llama 3.1 405B tokens per second number is the one measured against your prompt lengths, concurrency, and accuracy requirements. Use fp8 unless you have proven int4 works for your task, prefer hosts with batched fp8 kernels, and put a fallback gateway in front so a single degraded provider doesn’t take down your product. Speed is a distribution, not a slogan.