Llama 4 Maverick tokens per second is the metric most teams fixate on when choosing an inference provider, but the headline number rarely survives contact with production traffic. This analysis argues that cross-provider TPS comparisons are dominated by workload shape and infrastructure variables, not by the model itself, and that you should measure under your own conditions before trusting any ranking.
Why headline TPS lies
Providers publish impressive throughput numbers for Llama 4 Maverick, often measured on synthetic prompts with short contexts and saturated batch sizes. Those conditions do not match a real chat endpoint where request lengths vary, concurrency spikes, and KV caches compete for memory. A single average TPS hides the distribution. If a provider batches aggressively to hit high aggregate throughput but adds seconds of queue time, your users feel latency, not speed.
Worse, many public benchmarks report tokens per second at the GPU, not at the API boundary. Network serialization, load balancer hops, and tokenization overhead eat 10–20% before the bytes reach your client. When you compare Llama 4 Maverick tokens per second across providers, confirm whether the number includes time to first token or only generation phase.
The variables that dominate Llama 4 Maverick throughput
Quantization and weight layout
Llama 4 Maverick ships in multiple precisions. FP8 weights on H100s yield higher math throughput than FP16 on A100s, but some providers default to INT4 to save memory, trading slight quality for more concurrent sequences. The same model card can therefore post different Llama 4 Maverick tokens per second depending on the serving stack. If you need reproducibility, pin the precision in the request or via routing hints.
Batch size and concurrency
Throughput per token is a shared resource. At low concurrency, a single request might see modest TPS because the GPU is underutilized. At high concurrency, aggregate TPS climbs but per-request TPS drops. You cannot compare providers without fixing the concurrent request count. A provider that looks slow at one request per second may outperform at fifty because its scheduler is built for density.
Context length and KV cache pressure
Long system prompts allocate KV cache that persists across the generation. When cache pressure forces eviction or smaller batches, Llama 4 Maverick tokens per second degrades non-linearly. A provider with ample HBM or offload storage might keep large batches; one with tight memory limits will throttle. Test with your real system prompt length, not a toy 32-token input.
Provider load and autoscaling
Serverless endpoints scale cold. The first request after idle gets slower TPS than one after warm-up. Multi-tenant providers oversubscribe; your neighbor’s batch changes your numbers. This is why a Thursday-night benchmark differs from a Monday-morning one.
Measuring Llama 4 Maverick tokens per second correctly
Build a harness that mirrors your traffic. Below is a minimal streaming client against any OpenAI-compatible endpoint. It approximates token count by chunk deltas; in production, use the provider’s usage field or a local tokenizer.
import time, asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://provider.endpoint/v1", api_key="sk-...")
model = "meta-llama/llama-4-maverick"
async def sample():
start = time.perf_counter()
stream = await client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Summarize the tradeoffs of edge caching."}],
stream=True,
max_tokens=512,
)
chunks = 0
async for chunk in stream:
if chunk.choices[0].delta.content:
chunks += 1
elapsed = time.perf_counter() - start
return chunks / elapsed
async def main(concurrency=10):
results = await asyncio.gather(*[sample() for _ in range(concurrency)])
print(f"p50 TPS: {sorted(results)[len(results)//2]:.1f}")
asyncio.run(main())Run this at your target concurrency. Collect p50, p95, and p99 TPS, not just the mean. The p99 tells you what the worst user experiences.
What to record alongside TPS
Time to first token (TTFT) separates providers that batch from those that prioritize interactivity. Record batch size, input tokens, output tokens, and the region. Without these, a TPS number is meaningless. A provider reporting 70 Llama 4 Maverick tokens per second with 1.5s TTFT may be worse for chat than one at 45 TPS with 200ms TTFT.
Provider landscape without fake numbers
No two providers serve Llama 4 Maverick identically. Some expose the model on dedicated H100 clusters with speculative decoding; others on shared A100 pools with conservative batching. The qualitative pattern holds: dedicated hardware and fp8 quantization produce higher sustained Llama 4 Maverick tokens per second under load, while shared serverless tiers win on cold-start cost but lose on tail latency.
Speculative decoding impact
Providers that implement draft-model speculation can double decode speed on repetitive outputs. But speculation collapses on highly entropic text. If your traffic is code generation, you may see the published gain; if it’s creative writing, you won’t.
Region and network path
A provider with a datacenter near your users cuts TTFT even if its raw TPS is lower. Measure from your actual deployment zone, not from a benchmark rig in another continent.
Do not trust a static leaderboard. The same provider can shift its backend monthly as capacity changes.
Using a gateway to neutralize variability
An inference gateway like n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint, applies automatic fallback when a provider is rate-limited or degraded, and meters per-token usage. That lets you route Llama 4 Maverick traffic based on live TPS rather than a quarterly benchmark: if provider A slows, the gateway shifts to B without code changes. Honoring client routing directives and forwarding cache-control hints means your measurement loop can pin a provider when you need reproducible numbers, then release the pin in production.
This is the only sane way to consume cross-provider throughput data—treat it as a signal for routing, not a contract.
Tradeoffs: cost, TPS, and latency
Chasing maximum Llama 4 Maverick tokens per second usually increases cost per token or TTFT. Large batches amortize GPU cost but stall interactive UX. Smaller batches feel snappy but waste silicon.
If your app is a bulk summarizer, optimize for aggregate TPS and accept higher TTFT. If it’s a chat assistant, cap batch size to keep TTFT under 300 ms even if per-request TPS drops. Write this tradeoff into your SLOs before comparing providers.
Decisive takeaway
Stop comparing Llama 4 Maverick tokens per second as a static score. Stand up a measurement harness that replays your real prompt distribution and concurrency, collect p95 TPS and TTFT, and put a gateway in front that can shift providers on degradation. The provider that wins your benchmark today will change; the system that measures and routes continuously will not.