Llama 4 Scout inference speed benchmark

A credible Llama 4 Scout inference speed benchmark must isolate time-to-first-token from decode throughput, because the 17B active parameter MoE profile behaves nothing like a dense model under production load. Most published numbers blur those two phases, leading engineers to misconfigure timeouts and autoscaling. This analysis breaks down the variables that actually move the needle, shows how to measure without lying to yourself, and explains how to read third-party claims.

Why TTFT and TPS Diverge for MoE

Llama 4 Scout uses 17B active parameters out of 109B total spread across many experts. The prefill step still touches the full attention layers and the router, but each generated token only activates a subset of experts. That means prefill cost grows with sequence length roughly like a 109B dense model for the attention part, while decode cost per token looks closer to a 17B model.

A Llama 4 Scout inference speed benchmark that reports a single “tokens/sec” average hides this. You need two metrics: TTFT (prefill latency) and decode TPS (generation speed after first token).

Prefill cost scales with context

Long contexts kill TTFT. Scout supports very long windows; feeding 32K tokens of history forces the provider to compute attention over all of it before emitting token one. On H100-class GPUs, attention prefill for 32K tokens on a model this size can take hundreds of milliseconds to seconds depending on kernel efficiency and batch contention. If your traffic pattern is chat with rolling history, TTFT will track your context growth linearly until you prune or cache.

Decode cost scales with active params

Once generating, each step forwards only the active experts. Memory bandwidth, not compute, dominates. FP16 weights for 17B active params need ~34GB, fitting on a single H100 but leaving little headroom for KV cache. FP8 or INT4 quantization cuts memory traffic and raises TPS. The router itself adds negligible FLOPs, but expert load imbalance can cause tail latency if the scheduler doesn’t pack batches well.

Benchmark Methodology That Doesn’t Lie

Measure from the client edge. Network jitter matters, but it affects all providers equally if you run from the same region and same client instance. Use a streaming API and timestamp the first delta and subsequent deltas. Never use the non-streaming create call and divide total time by token count; that folds TTFT into decode and masks the phase that breaks UX.

Control these variables:

• Input length (fixed at 512, 4K, 32K)
• Output max tokens (fixed at 256)
• Batch size (concurrency 1, 8, 32)
• Quantization (ask provider or assume FP16 unless documented)

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.example.com/v1", api_key="key")

def measure(prompt: str, max_tokens=256):
    t0 = time.perf_counter()
    stream = client.chat.completions.create(
        model="llama-4-scout",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )
    ttft = None
    n_tokens = 0
    t_first = None
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            if ttft is None:
                ttft = time.perf_counter() - t0
                t_first = time.perf_counter()
            else:
                n_tokens += 1
    total = time.perf_counter() - t_first
    tps = n_tokens / total if total > 0 else 0
    return ttft * 1000, tps

Run at least 20 iterations per configuration to get p50 and p99. Warm up the provider with five discarded requests so you don’t measure cold autoscalers.

Sample result shape

{
  "provider": "A",
  "input_tokens": 512,
  "output_tokens": 256,
  "ttft_p50_ms": 220,
  "ttft_p99_ms": 410,
  "decode_tps_p50": 68,
  "decode_tps_p99": 52
}

A Llama 4 Scout inference speed benchmark should publish this breakdown, not a blended number. If a post shows only “72 tokens/sec”, assume they measured a 32-token input and 256-token output and threw away the first 200ms.

Provider Infrastructure Differences

GPU class is the obvious lever. H100 and B200 deliver different decode bandwidth. But software stack matters more than silicon: vLLM, TensorRT-LLM, and SGLang implement expert parallelism differently. A provider running naive Megatron may exhibit 30% lower TPS than one with fused MoE kernels and continuous batching.

Quantization and caching

FP8 reduces weight footprint and increases TPS, but some providers keep FP16 for quality. Prompt caching (reusing KV for repeated system prompts) slashes TTFT for subsequent requests. If your benchmark doesn’t send cache_control hints, you penalize providers that would otherwise cache.

When using a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models and honors client routing directives, you can forward provider cache-control hints unchanged while rotating providers via headers to collect comparable data. The measurement code stays identical; only the routing header changes.

Fallback skews results

Gateways with automatic fallback when a provider is rate-limited or degraded are great for production but must be bypassed during benchmarking. If provider A fails and the gateway silently reroutes to provider B, your p99 suddenly mixes two infrastructures. Disable fallback or pin the upstream explicitly.

What the Benchmark Reveals About Tradeoffs

Higher concurrency destroys TTFT if the provider packs batches greedily. At batch 32, TTFT may spike 3x while aggregate TPS rises. For interactive chat, prioritize TTFT under your real concurrency. For batch extraction, maximize total TPS at high batch.

Latency vs cost

Cheaper endpoints often throttle batch size or use older GPUs. A Llama 4 Scout inference speed benchmark that only measures speed misses the $/million token dimension. Compute cost per token is roughly proportional to active params and quantization, but provider markup varies widely. Plot TPS against price per million output tokens to find the efficient frontier.

Context length impact

Doubling input from 4K to 32K can triple TTFT but leave decode TPS flat. If your app sends long RAG contexts, optimize prefill via chunked caching or truncate aggressively. The decode phase doesn’t care about input length after prefill completes.

Common Measurement Bugs

• Using wall-clock from request send without accounting for DNS/TLS handshake. Use a persistent client.
• Not excluding queue time when provider is saturated. If you see TTFT > 2s at concurrency 1, the provider is likely overloaded or cold.
• Mixing streaming and non-streaming calls across runs.
• Ignoring tokenization differences: the tokenizer may split your synthetic padding differently than real text, altering true token counts.
• Sampling temperature > 0 doesn’t affect speed materially, but top-p filtering can add tiny overhead; keep it fixed.

Interpreting Third-Party Benchmarks

Many public Llama 4 Scout inference speed benchmark posts use synthetic short prompts. They show impressive TPS that collapses with 8K context. Look for the input distribution. If the author doesn’t state input length, treat the number as best-case marketing. An honest post will show curves: TTPS vs input length, TTFT vs input length, and p99 under concurrency.

How to Simulate Load Realistically

Use a replay of real traffic. If you don’t have it, generate prompts with varying lengths from a Zipf distribution to mimic natural skew.

import random, string

def zipf_prompt(alpha=1.2, max_len=8000):
    length = min(max_len, int(random.zipf(alpha)) + 50)
    return " ".join(random.choices(string.ascii_lowercase, k=length//5))

Drive concurrency with a worker pool, not sleep-based loops. Capture per-request TTFT and TPS, then aggregate percentiles. A Llama 4 Scout inference speed benchmark run this way surfaces the difference between a provider tuned for bursty chat and one tuned for bulk processing.

Decisive Takeaway

Run your own Llama 4 Scout inference speed benchmark with separated TTFT and decode TPS at fixed input lengths and concurrencies. Ignore blended numbers. For latency-sensitive products, choose a provider with FP8 decode and prompt caching, keep batch size per replica low, and set client timeouts off your p99 TTFT plus generation estimate. For throughput jobs, push concurrency and accept TTFT variance. The model architecture rewards systems that treat prefill and decode as distinct problems—engineers who measure both will ship faster than those who trust a single speed score.