Llama 4 Maverick throughput benchmark by region

Most teams assume that renting the same GPU SKU in two clouds yields identical inference speed. The reality of Llama 4 Maverick throughput by region is that network proximity and provider queue depth distort the picture far more than silicon revisions, and a naive single-region benchmark will mislead your capacity plan.

Why region is the hidden variable in MoE serving

Llama 4 Maverick is a 400B-parameter mixture-of-experts model with 17B active parameters per token. Decode throughput for a single request is governed by memory bandwidth to load expert weights, not by tensor core flops. When providers stand up Maverick in a region, they choose GPU type (H100, H200, or older A100), host count, and batching policy. Those choices differ by region based on allocation priority and datacenter vintage.

A region with H200s and aggressive continuous batching will sustain higher tokens/sec under concurrency than a region with A100s and conservative scheduling. But the gap shrinks at low concurrency because a single request cannot saturate the memory bus.

Regional placement also affects expert sharding. Maverick’s 128 experts are typically spread across multiple GPUs with tensor parallelism. If a provider uses TP=8 in one region and TP=4 in another, cross-node traffic for expert routing changes, which subtly impacts per-token latency even on the same GPU model.

Benchmark methodology that doesn’t lie

You cannot compare regions by firing one curl request. You need to hold constant: prompt length, max tokens, concurrency, and sampling params. Use an OpenAI-compatible client and loop with timing.

import asyncio, time, openai

async def bench(client, region, concurrency, prompt, max_tokens=256):
    async def one():
        t0 = time.perf_counter()
        stream = await client.chat.completions.create(
            model="meta-llama/llama-4-maverick",
            messages=[{"role":"user","content":prompt}],
            max_tokens=max_tokens,
            stream=True,
            extra_headers={"x-region": region}  # provider routing hint
        )
        tokens = 0
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                tokens += 1
        return time.perf_counter() - t0, tokens
    tasks = [one() for _ in range(concurrency)]
    results = await asyncio.gather(*tasks)
    total_t = sum(r[0] for r in results)
    total_tok = sum(r[1] for r in results)
    return total_tok / total_t, results

client = openai.AsyncOpenAI(base_url="https://api.example.com/v1", api_key="key")

Run this against each region tag your provider supports. Capture p50 and p99 latency, not just average throughput. When you plot Llama 4 Maverick throughput by region from this data, you will see variance that tracks time of day in that region’s local business hours more than it tracks hardware specs.

Keep your prompt representative: a 1k-token RAG context exercises KV cache allocation differently than a 50-token chat message. Run each region for at least five minutes at your target concurrency to absorb scheduler warm-up.

Time to first token vs decode throughput

Two metrics matter: TTFT (time to first token) and decode rate (tokens/sec after first token). TTFT is dominated by network round trip plus queue wait. Decode rate is dominated by GPU memory bandwidth.

In repeated runs, TTFT in us-east-1 was consistently lower than in ap-south-1 for clients located in North America, owing to shorter cable paths and fewer transit hops. Once streaming started, decode rates on identical GPU classes were close, because the decode loop is local to the GPU and unaffected by wide-area latency.

That means Llama 4 Maverick throughput by region looks dramatically different if you weight TTFT heavily (chat UX) versus if you only care about bulk generation cost. A batch job tolerant of 2-second startup will see flat regional performance; an interactive agent will not.

Prompt cache effects

Providers that honor cache-control hints can skip recomputing prefix KV caches. A region where your system prompt is already cached shows markedly lower TTFT. If your client sends cache_control blocks, confirm the regional endpoint respects them—some providers only enable caching in flagship regions.

The saturation factor

Providers oversubscribe. A region that launched recently may have spare capacity; a region hosting a popular frontier model may be saturated during local peak. Saturation hits TTFT first, then degrades decode because batching gets tighter and preemption kicks in.

We observed that during EU business hours, eu-central-1 showed TTFT spikes while decode held steady. That is a scheduling signal, not a hardware limit. The same region at 3am local time behaved like a different machine.

This is why a one-time benchmark is worthless. You need a rolling measurement or at least samples across two time windows.

Tradeoffs of pinning a region

You can pin a region to get predictable latency. The cost is fragility: a single provider incident or capacity crunch leaves you dark. Multi-region failover sounds easy but complicates client logic when streams are partial and responses must be stitched.

Another tradeoff is data residency. Some stacks must keep prompts in eu-west. That constraint may force a region with worse throughput; measure it before assuming you need to negotiate for better hardware or file a support ticket.

Client-side fallback sketch

regions = ["us-east-1", "eu-west-1", "ap-southeast-1"]
async def call_with_fallback(prompt):
    for r in regions:
        try:
            return await bench(client, r, 4, prompt)
        except openai.APIStatusError as e:
            if e.status_code in (429, 503):
                continue
            raise
    raise RuntimeError("all regions degraded")

This naive loop adds tail latency because each failure burns a full request timeout. A server-side router absorbs that cost.

Interpreting tokens/sec correctly

Do not confuse per-request decode rate with aggregate system throughput. A provider may report “high throughput” by packing 64 concurrent requests into one batch, which improves GPU utilization but raises your individual TTFT. Always benchmark at your own concurrency, not the provider’s max.

Also note that Maverick’s MoE sparsity means longer sequences do not scale memory linearly per token—experts are reused. Regions with larger aggregate HBM per node can hold bigger batches, so their throughput curve stays flat longer under rising concurrency.

Routing around degraded zones

A gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is rate-limited, lets you stop writing region loops. You send one request, set a routing directive, and the gateway honors it or falls back to a healthy region. It also forwards provider cache-control hints so your prefix caches survive region switches.

That said, a gateway cannot defeat physics: if your client is in Sydney and the only healthy region is Virginia, TTFT will still hurt. Use client-side geo-DNS or edge proxies to colocate, and treat the gateway as a resilience layer rather than a latency cure.

Decisive takeaway

Benchmark every region you can reach with the same concurrency profile and prompt shape before you trust any single number. The honest reading of Llama 4 Maverick throughput by region is that it is a function of proximity plus provider load, not a fixed property of the model. Pin to the region closest to your users for TTFT, but build fallback to at least one other region for resilience, and measure continuously because saturation patterns drift week to week.