n4nAI
Open console →
Benchmarks & performanceAnalysisLong-Context Latency Benchmarks

Llama 4 Scout long-context latency at 10M tokens

Analysis of Llama 4 Scout 10M context latency: why KV-cache footprint and prefill scheduling dominate, and how to serve open-weight long-context models in production.

n4n Team4 min read906 words

Audio narration

Coming soon — every post will get a voice note here.

Llama 4 Scout 10M context latency is the real barrier between demoing a 10-million-token open-weight model and shipping it as a reliable service. This analysis argues that the dominant cost is not floating-point operations but KV-cache footprint and prefill scheduling, and that teams who treat long context as just a bigger prompt will see tail latency explode non-linearly.

The thesis: latency is a systems problem

Llama 4 Scout ships as an open-weight MoE with a declared 10M-token window. The weights are fixed; the serving stack decides whether a request returns in seconds or stalls for minutes. Decompose response time into prefill (processing input), decode (generating output), and the overhead of moving KV tensors across devices. At 10M tokens, the last of these dwarfs the others if unmanaged.

The mistake most teams make is benchmarking latency on a 2K-token prompt and extrapolating. Long context changes the shape of the problem: memory capacity, not compute throughput, becomes the binding constraint.

What 10M tokens does to the KV cache

Take a 32-layer transformer with 4096 hidden dimensions in fp16. Per token, per layer, the KV cache stores two tensors of shape [hidden]. That is 2 * 4096 * 2 bytes = 16 KB per token per layer. Across 32 layers, 512 KB per token. At 10M tokens, that is 5.12 TB of KV state.

layers = 32
hidden = 4096
bytes_per_fp16 = 2
kv_per_token = layers * 2 * hidden * bytes_per_fp16
total_tb = kv_per_token * 10_000_000 / 1e12
print(total_tb)  # 5.12 TB

No single server holds that in HBM. You must shard across nodes, page to host memory, or both. This linear growth is the first reason Llama 4 Scout 10M context latency scales poorly under naive deployments. The cache must be allocated before prefill; if your scheduler fragments memory, you refuse requests long before compute saturates.

GQA and architectural mitigations

Scout uses grouped-query attention (GQA), which shrinks KV by sharing heads across groups. That cuts the 5.12 TB figure by the GQA ratio (often 8x), bringing it to ~640 GB—still far beyond a single 8-GPU node’s HBM. Latent attention or sliding-window hybrids reduce active KV per step but do not eliminate the need to store the full sequence for backward attention. The cache is unavoidable.

Prefill: where the seconds disappear

Prefill computes attention over the full prompt. With flash-attention kernels, the quadratic cost is partially hidden, but you still move hundreds of GB of KV through the interconnect. On an 8x H100 node with ~3.35 TB/s aggregate HBM bandwidth, streaming 640 GB once takes >0.19 seconds minimum, before any math. In practice, chunked prefill and pipeline bubbles push TTFT to tens of seconds for a single 10M-token request.

The Llama 4 Scout 10M context latency during prefill is therefore bounded by memory bandwidth and parallelization strategy, not parameter count. MoE helps: only a subset of experts activate per token, reducing compute but not KV size. If you skip chunked prefill, the scheduler blocks decode for all other requests until the 10M prompt is processed—a latency disaster.

# vLLM with tensor-parallel 8 and chunked prefill
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-10M \
  --tensor-parallel-size 8 \
  --max-model-len 10000000 \
  --enable-chunked-prefill \
  --kv-cache-dtype fp8

That flag set will not make latency vanish, but it prevents a single request from monopolizing the GPU.

Decode phase and time-to-first-token

After prefill, decode generates tokens one step at a time. The KV cache is read every step. With 640 GB resident, a single decode step fetches a slice per layer. If the cache is sharded poorly, all-to-all communication dominates. Throughput per request collapses if you batch multiple 10M-context requests because aggregate KV exceeds fabric capacity.

Time-to-first-token (TTFT) is prefill-dominated. Time-per-output-token (TPOT) is decode-dominated and stays roughly constant per token, but the absolute TPOT degrades when KV fetches contend with weight reads.

Memory bandwidth and batching tradeoffs

You can trade latency for throughput by batching. Each additional 10M-context sequence multiplies KV footprint. A single node caps at maybe one or two such sequences. To serve many, you need disaggregated prefill/decode or layered caching. The Llama 4 Scout 10M context latency for a batched system is the sum of queue time and per-sequence prefill, not the max of independent runs.

Prefix caching across requests

If many users send the same long system prompt or document, prefix caching reuses KV. OpenAI-compatible gateways that forward provider cache-control hints make this explicit. When routing across providers, an endpoint that honors client routing directives lets you pin Scout to a cluster sized for 10M without code changes. n4n.ai exposes such an OpenAI-compatible gateway across 240+ models, which is useful for A/B testing latency under different backends.

Measuring it yourself

Instrument TTFT and TPOT separately. A minimal client call:

import time, openai
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
t0 = time.time()
stream = client.chat.completions.create(
    model="meta-llama/llama-4-scout-10m",
    messages=[{"role":"user","content": BIG_INPUT}],
    stream=True,
    extra_body={"provider": {"data_collection":"deny"}}
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        t1 = time.time()
        break
print("TTFT", t1-t0)

This isolates prefill cost from generation. Run it against a single-sequence deployment to get the floor; then add concurrency to see the cliff.

Tradeoffs: when not to use 10M context

If your task is retrieval over a fixed corpus, a vector DB with 8K context beats a 10M prompt on cost and latency. The Llama 4 Scout 10M context latency penalty is justified only when the model must attend to cross-document relationships that sparse retrieval misses—e.g., reconciling contradictory clauses across 50 contracts. Otherwise you burn memory for negligible accuracy gain.

Cost of ignoring the cliff

A team that sets max_model_len=10M globally will see sporadic 503s when the KV allocator fragments. The fix is not bigger GPUs; it is admission control. Reject prompts above a per-route threshold or summarize upstream.

Takeaway

Treat Llama 4 Scout 10M context latency as a distributed systems metric. Size KV cache first, schedule prefill with chunking, and cap concurrency per node. Open-weight long context is production-viable only with deliberate sharding, prefix caching, and eviction—not by scaling model replicas blindly. If you cannot measure TTFT under your real batch size, you do not yet understand your latency.

Tagsllama-4-scoutlong-context10m-tokensopen-weight

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceHow context length affects time to first tokenAnalyze how prompt size drives time to first token in LLM inference, why it's sublinear in practice, and how to benchmark and reduce long-context TTFT.Benchmarks & performanceHow much does a 100k-token prompt add to response timeAnalyzing how a 100k token prompt affects LLM response time: prefill cost, time-to-first-token, caching, and practical latency tradeoffs for engineers.Benchmarks & performanceLong-context latency: Claude Opus 4.5 at 200k tokensAnalyzing Claude Opus 4.5 200k context latency: prefill dominates TTFT, prefix caching helps, and full context isn't always worth the cost.Benchmarks & performanceMeasuring latency cost of large system promptsStep-by-step method to measure system prompt latency cost on LLM endpoints, isolating prefill overhead with streaming and controlled prompt sizes across models.Benchmarks & performanceWhy long-context requests slow down time to first tokenAnalyzes why long context time to first token slowdown occurs, breaking down prefill cost, attention scaling, and scheduling, with practical mitigations.Benchmarks & performance32k vs 128k vs 1M: context length latency scalingContext length latency scaling from 32k to 1M tokens is non-linear. This analysis explains the mechanics and gives engineers concrete mitigation strategies.

All long-context latency benchmarks posts →