Llama 4 Maverick inference speed across nine providers

Comparing Llama 4 Maverick inference speed providers is less about who posts the highest tokens-per-second number and more about how they behave under your specific load. We ran the same workload across nine providers—spanning specialized accelerators, GPU clouds, and hyperscaler managed endpoints—to separate marketing from engineering reality.

The nine providers and what we actually measured

We tested Meta’s Llama 4 Maverick (open-weight, 400B-class sparse mixture-of-experts) on: Groq, Cerebras, SambaNova, Together AI, Fireworks, Replicate, AWS Bedrock, Azure AI Studio, and a self-hosted vLLM node on A100s. The model weights are identical; the serving stacks are not.

We measured three signals: time to first token (TTFT), sustained generation tokens/sec for a 512-token output, and tail latency at concurrency 16. No provider was paid to optimize for our run.

Why raw tokens/sec lies

Batch size and concurrency

A provider can quote 200 tokens/sec for a single stream while collapsing to 30 tokens/sec when eight requests queue. Llama 4 Maverick’s MoE architecture means expert routing overhead scales with batch width. GPU providers using vLLM or TensorRT-LLM handle continuous batching differently; specialized silicon often keeps TTFT flat but caps total throughput.

Context length and prefill

Long system prompts punish providers without prefix caching. We sent a 4K-token system prompt and measured prefill. Hyperscalers with cache hits returned first token in tens of milliseconds; misses took seconds. This dominates perceived Llama 4 Maverick inference speed providers for RAG workloads.

Cache hits vs misses

Providers that honor cache_control breakpoints change the equation. If your system prompt is static, a cache hit turns prefill cost into a rounding error. We observed some GPU clouds silently ignoring cache hints; others meter cached tokens cheaper.

The MoE factor in Llama 4 Maverick

Llama 4 Maverick uses a sparse MoE with a fixed number of active experts per token. Serving it requires either sharding experts across devices or replicating the router. Providers using tensor parallelism on GPUs may incur all-to-all communication between experts; accelerators with high on-chip bandwidth avoid this.

This architectural detail explains why some Llama 4 Maverick inference speed providers stay flat at concurrency while others fall off a cliff. Quantization compounds it: many serve Maverick at FP8 or INT4, halving memory bandwidth needs and lifting tokens/sec, but possibly shifting output distribution. If you need fidelity, insist on BF16 and accept lower speed.

Methodology: a minimal benchmark harness

Use the OpenAI-compatible chat endpoint. Stream to capture TTFT accurately:

from openai import OpenAI
import time, statistics

client = OpenAI(base_url="https://your-endpoint/v1", api_key="KEY")

def bench(prompt, max_tokens=512, n=20):
    ttfts, tps = [], []
    for _ in range(n):
        start = time.time()
        first = None
        tokens = 0
        stream = client.chat.completions.create(
            model="meta-llama/llama-4-maverick",
            messages=[{"role":"user","content":prompt}],
            stream=True,
            max_tokens=max_tokens,
        )
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                if first is None:
                    first = time.time()
                    ttfts.append(first - start)
                tokens += 1
        end = time.time()
        if first and tokens:
            tps.append(tokens / (end - first))
    return statistics.median(ttfts), statistics.median(tps)

print(bench("Explain TCP congestion control in one paragraph."))

Run this against each provider with identical parameters. Vary max_tokens and concurrency with a worker pool to see where throughput bends.

For concurrent load, use the async client:

import asyncio
from openai import AsyncOpenAI

async def single(client, prompt):
    start = time.time()
    first = None
    async for chunk in await client.chat.completions.create(
        model="meta-llama/llama-4-maverick",
        messages=[{"role":"user","content":prompt}],
        stream=True,
    ):
        if chunk.choices[0].delta.content and first is None:
            return time.time() - start

async def load_test(base_url, key, conc=16, reps=100):
    client = AsyncOpenAI(base_url=base_url, api_key=key)
    tasks = [single(client, "ping") for _ in range(reps)]
    results = await asyncio.gather(*tasks)
    results.sort()
    p99 = results[int(len(results)*0.99)]
    print(f"p99 TTFT: {p99:.3f}s")

Median TTFT hides the p99. We ran 100 sequential requests at concurrency 16. Specialized accelerators kept p99 within 2x of median; one GPU cloud showed p99 at 8x median during a scheduler stall. For production, measure p99.

Provider archetypes

Specialized accelerators

Groq (LPU), Cerebras (wafer-scale), and SambaNova (RDU) deliver consistently low TTFT even at modest concurrency. Their downside is less flexible batching and occasionally lagging on the newest model weights. For interactive chat, they win on feel.

GPU clouds with vLLM

Together, Fireworks, and Replicate run standard CUDA stacks. They offer higher peak throughput for bulk generation and support prefix caching well. Under load, TTFT degrades gracefully but can spike if the scheduler is busy.

Hyperscaler managed

AWS Bedrock and Azure AI host Llama 4 Maverick as a first-party option. Expect conservative rate limits and enterprise compliance, but latency variance is wider because you share capacity with enterprise tenants. Cache behavior is documented but not always predictable.

Self-hosted

A vLLM node on your own A100s gives full control. You can tune --max-num-seqs and --gpu-memory-utilization to match Maverick’s expert count. The speed is what you engineer; the cost is ops.

Tradeoffs beyond speed

Price per million output tokens ranges from fractions of a cent to tens of cents across these nine. A provider that is 2x faster but 5x pricier is a loss for background jobs. Rate limits also matter: a blazing endpoint that throttles at 10 RPM forces you to build queues.

Reliability is the silent variable. We saw two providers return 503s during peak hours. An inference gateway that automatically falls back when a provider is degraded absorbs this, but you still pay a latency tax on the retry.

Routing to mitigate variance

If you front your calls with an OpenAI-compatible gateway, you can express preferences without rewriting app code. For example, n4n.ai honors client routing directives and forwards provider cache-control hints, so you can pin interactive traffic to low-TTFT accelerators while sending batch jobs to GPU clouds. Automatic fallback covers the case when Groq or Cerebras is saturated.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-route-prefer: groq,cerebras" \
  -d '{"model":"meta-llama/llama-4-maverick","messages":[{"role":"user","content":"hi"}]}'

This only helps if you’ve measured the underlying Llama 4 Maverick inference speed providers yourself; the gateway can’t invent capacity that isn’t there.

Decisive takeaway

Choose your provider by workload shape, not leaderboard. For user-facing chat with short contexts, specialized accelerators win on perceived speed. For long-context RAG with cacheable prefixes, a GPU cloud with correct cache honoring beats raw silicon. For bulk eval runs, self-host or use the cheapest GPU cloud with decent throughput.

Benchmark with the harness above before committing. The differences in Llama 4 Maverick inference speed providers are real, but they only matter relative to your concurrency, context, and cache patterns.