Together AI Llama 4 Maverick throughput benchmarks

The Together AI Llama 4 Maverick throughput benchmark numbers circulating in vendor docs look impressive, but they measure a narrow slice of production reality. Llama 4 Maverick is a 400B-parameter mixture-of-experts model with 17B active parameters per token, and serving it efficiently requires careful batching and expert parallelism. This analysis breaks down what those benchmarks actually capture, how to reproduce meaningful numbers, and where the tradeoffs bite.

Why throughput benchmarks for MoE models mislead

Llama 4 Maverick uses 128 experts with 17B active parameters per forward pass. That architecture decouples parameter count from per-token compute, but it does not eliminate memory bandwidth pressure during decode. Throughput in tokens/sec is therefore a function of how well the serving stack pipelines expert routing and all-to-all communication.

A Together AI Llama 4 Maverick throughput benchmark that reports a single peak number typically fixes batch size to saturate the GPU and uses short prompts. That hides the latency cost at batch size one, which is what interactive apps see.

The expert routing tax

Each token activates a subset of experts. In a naive implementation, this forces a gather-scatter across devices. Together’s stack uses custom CUDA kernels and possibly expert sharding to amortize that cost, but the tax grows with sequence length because the router runs every step.

If you benchmark only 128-token inputs, you measure warmup-dominated runs. Production traffic with 4k context shifts the ratio toward compute-bound decode, lowering tokens/sec per request.

Anatomy of a Together AI Llama 4 Maverick throughput benchmark

Vendor benchmarks usually specify: model ID, hardware (H100/H200), batch size, input length, output length, and concurrency. They omit the system prompt padding, the request scheduling policy, and whether they count prefill tokens as throughput.

Test harness essentials

You need a loop that sends N concurrent requests, measures time to first token and completion, and computes total output tokens / wall clock. Use the OpenAI-compatible endpoint Together exposes.

import openai, time

client = openai.OpenAI(
    base_url="https://api.together.xyz/v1",
    api_key="TOGETHER_KEY"
)

def complete(prompt, max_tokens=256):
    start = time.monotonic()
    resp = client.chat.completions.create(
        model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
        messages=[{"role":"user","content":prompt}],
        max_tokens=max_tokens,
        stream=False
    )
    elapsed = time.monotonic() - start
    return resp.usage.completion_tokens, elapsed

A honest Together AI Llama 4 Maverick throughput benchmark will show a knee where adding concurrency stops helping because expert lanes saturate.

Controlling for context length and batch size

Fix input tokens at your p95 production value. Vary concurrency from 1 to 64. Plot tokens/sec per request and aggregate throughput. Use a fixed payload:

{
  "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
  "messages": [{"role":"user","content":"Summarize: <4k chars>"}],
  "max_tokens": 512,
  "temperature": 0
}

Short-input runs overstate throughput because prefill is cheap and the batch scheduler stays in decode-dominated mode.

Tradeoffs: throughput vs latency vs cost

High throughput almost always means larger batches, which raises time-to-first-token (TTFT). For a chat UI, TTFT > 500ms feels broken even if aggregate tokens/sec is 30k. Together’s optimized kernels reduce per-step overhead, but the fundamental scheduling conflict remains.

Cost tracks throughput inversely: if a benchmark shows 2x tokens/sec per GPU, you need half the GPUs for the same traffic. But if you only have sparse traffic, you pay for reserved capacity anyway.

Where the MoE wins

Dense 400B would require ~8 H100 just to hold weights in FP8; Maverick’s 17B active means each token loads less data. That translates to higher decode throughput per wafer than a dense peer, provided the router is balanced. Imbalanced experts cause tail latency spikes that a single-number benchmark masks.

Streaming changes the math

If you stream responses, the client sees tokens earlier, but the server-side generation cost is identical. A vendor-published Together AI Llama 4 Maverick throughput benchmark that counts streamed tokens as “delivered” while ignoring TTFT is measuring network pipelining, not model efficiency.

How to run your own benchmark

Don’t trust a screenshot. Stand up a harness that mimics your traffic shape.

Minimal Python harness

import openai, time, threading

client = openai.OpenAI(base_url="https://api.together.xyz/v1", api_key="KEY")
PROMPT = "Explain mixture-of-experts routing in 200 words."

def worker(results, max_tokens=200):
    t0 = time.monotonic()
    r = client.chat.completions.create(
        model="meta-llama/Llama-4-Maverick-17B-128E-Instruct",
        messages=[{"role":"user","content":PROMPT}],
        max_tokens=max_tokens
    )
    dt = time.monotonic() - t0
    results.append((r.usage.completion_tokens, dt))

def bench(concurrency, requests):
    threads = []
    results = []
    for i in range(requests):
        t = threading.Thread(target=worker, args=(results,))
        threads.append(t)
        if len(threads) >= concurrency:
            for x in threads: x.start()
            for x in threads: x.join()
            threads = []
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    total_tok = sum(t for t,_ in results)
    total_time = max(dt for _,dt in results)
    print(f"{concurrency} conc: {total_tok/total_time:.1f} tok/s")

bench(1, 10)
bench(8, 80)

This is crude but shows the shape. Real harness uses async and percentiles.

Interpreting results

Look at p50 and p99 TTFT separately from tokens/sec. A Together AI Llama 4 Maverick throughput benchmark that only reports mean tokens/sec hides the p99 blowup at high concurrency.

If you front your calls with an OpenAI-compatible gateway like n4n.ai, you get per-token metering, automatic fallback when a provider is degraded, and it forwards provider cache-control hints—but the raw Together AI Llama 4 Maverick throughput benchmark you measure is identical to calling Together directly.

Comparing providers without fabricating numbers

When you evaluate Together against other Llama 4 hosts, run the identical harness against each. Differences in prefill batching will surface quickly. Some providers cache KV across requests; Together’s cache behavior can be controlled via headers if you use a proxy that forwards them.

Do not publish cross-provider numbers unless you pin model revision, tensor parallelism, and context. The phrase “Together AI Llama 4 Maverick throughput benchmark” should refer to a specific hardware and software commit, or it is meaningless.

Decisive takeaway

Together’s serving stack extracts strong throughput from Llama 4 Maverick’s MoE design, but vendor benchmarks are optimized for headline numbers. Build your own harness, fix your real context length, and measure p99 latency alongside tokens/sec. If your traffic is bursty and latency-sensitive, you will trade some peak throughput for responsiveness; if it’s bulk processing, Together’s batch efficiency is hard to beat. Choose based on that curve, not a screenshot.