n4nAI
Open console →
Benchmarks & performanceAnalysisLlama 4 Inference Speed by Provider

Llama 4 inference speed benchmark for long context

Analyzing Llama 4 inference speed long context: why prefill and decode must be measured separately, and how provider batching and KV cache shape real latency.

n4n Team4 min read864 words

Audio narration

Coming soon — every post will get a voice note here.

The headline claim that a model serves tokens at 40 tps hides more than it reveals. This analysis of Llama 4 inference speed long context argues that long-context performance splits into two distinct phases—prefill and decode—and that provider-side engineering choices dominate the numbers you’ll see in production. If you benchmark only aggregate throughput, you will mis-size your infrastructure and ship a chat UI that feels broken on the first long document.

Why long context breaks naive latency models

A transformer processes a prompt by building a key/value cache for every token. Even with FlashAttention, the prefill pass performs compute proportional to prompt length × hidden size, and the KV cache grows linearly with context. Llama 4 uses grouped-query attention, which shrinks the cache relative to standard MHA, but a 100k-token input still forces several gigabytes of state per request instance.

During decode, each generated token requires a read of the entire KV cache. Memory bandwidth, not FLOPs, becomes the ceiling. Double the context length and you roughly double the cache fetch per token, directly cutting inter-token latency (ITL) on bandwidth-bound hardware.

Naive latency models assume constant tps regardless of input size. That assumption dies at 32k tokens.

Prefill vs decode: the two speeds you actually care about

Treat these as separate benchmarks:

  • Time to first token (TTFT): dominated by prefill. Sensitive to prompt length, batching, and attention kernel efficiency.
  • Inter-token latency (ITL): dominated by decode. Sensitive to KV cache size, quantization, and batch contention.

A provider can post great decode numbers on a 1k prompt and fall apart at 128k because prefill queues behind other requests. Measure both.

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
model = "meta/llama-4-70b"

prompt = open("long_doc.txt").read()  # ~100k tokens
messages = [{"role": "user", "content": prompt}]

start = time.perf_counter()
stream = client.chat.completions.create(
    model=model, messages=messages, stream=True,
    extra_body={"cache_control": {"type": "ephemeral"}}  # forwarded to provider
)
first_token_ts = None
tok_count = 0
for chunk in stream:
    if chunk.choices[0].delta.content:
        if first_token_ts is None:
            first_token_ts = time.perf_counter()
            ttft_ms = (first_token_ts - start) * 1000
        tok_count += 1
end = time.perf_counter()
decode_ms = (end - first_token_ts) * 1000
print(f"TTFT: {ttft_ms:.0f}ms, decode total: {decode_ms:.0f}ms, tokens: {tok_count}, ITL avg: {decode_ms/tok_count:.1f}ms")

The script isolates TTFT from average ITL. Run it at multiple prompt lengths to see where the curve bends.

Provider variables that skew Llama 4 inference speed long context

When you compare Llama 4 inference speed long context across providers, the same model weights can show 5–10× TTFT variance. The causes are systemic, not magical.

Continuous batching

Providers without continuous batching pad batches to the longest sequence, wasting compute on short requests behind a 128k prompt. Those with it (e.g., based on Triton or vLLM schedulers) interleave new requests into free slots, keeping TTFT predictable under load. Ask the provider for their scheduler architecture; if they can’t answer, assume the worst.

Prefix caching

If you send the same system prompt or document prefix repeatedly, a provider that caches KV slices across requests turns your second call into a decode-only operation. This collapses TTFT from seconds to milliseconds. The gateway should forward your cache-control hints:

{
  "model": "meta/llama-4-70b",
  "messages": [{"role": "system", "content": "LONG_POLICY_DOC"}],
  "cache_control": {"type": "ephemeral"}
}

Without that forwarding, your optimization dies at the proxy.

Quantization

FP8 or INT4 weights reduce memory traffic and raise decode tps, but some providers apply aggressive quantization only to smaller Llama 4 variants. Verify whether the 70B long-context endpoint is actually quantized, or if you’re paying for BF16 latency.

Context parallelism

At 256k+ context, a single GPU may not hold the cache comfortably. Providers that shard the context across GPUs (tensor or sequence parallel) keep TTFT sane but add all-reduce overhead. This trades latency for feasibility.

Benchmarking methodology that doesn’t lie

Build a length ladder: 4k, 16k, 32k, 64k, 128k tokens. Use real documents, not repeated “foo”, because repetitive tokens compress under attention and misrepresent cache pressure.

For each length, measure:

  1. Cold TTFT (no prefix cache)
  2. Warm TTFT (with cache hint)
  3. ITL under load (8 concurrent streams)

A minimal curl for a single cold measurement:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"meta/llama-4-70b","messages":[{"role":"user","content":"'"$(cat long.txt)"'"}],"stream":true}'

Timestamp the first byte versus the request start. Then parse the SSE stream for token gaps.

Ignore providers that only publish “tokens/sec” without specifying context length. That number is for 1k prompts and is irrelevant to your RAG pipeline.

Tradeoffs: when to truncate context

Long context is a crutch. Feeding 128k tokens to Llama 4 avoids building a retriever but costs prefill seconds per call and multiplies GPU spend. In one internal test, moving from 128k to a 24k retrieved subset cut TTFT by ~4× and improved answer accuracy because the model attended to relevant spans instead of noise.

The decision rule: if your accuracy at 32k with retrieval matches 128k without, truncate. The Llama 4 inference speed long context penalty is not linear—past 64k, TTFT often grows faster than prompt length due to cache eviction and HBM pressure.

Routing and fallback realities

An OpenAI-compatible gateway such as n4n.ai that honors routing directives lets you pin a provider with strong long-context numbers while keeping automatic fallback when that provider is degraded. But the benchmark responsibility stays with you: fallback routes may have different TTFT profiles, and your users will feel the switch.

If you send cache_control hints, confirm the gateway forwards them. N4n.ai’s endpoint addresses 240+ models and forwards provider cache-control hints; use that to keep warm prefixes intact across route changes.

Decisive takeaway

Stop quoting single-number tps for Llama 4. Measure TTFT and ITL separately across a context ladder, demand continuous batching and prefix caching from your provider, and truncate aggressively when retrieval can cover the gap. The engineers who win on Llama 4 inference speed long context are those who treat prefill as a capacity planning problem and decode as a bandwidth budget—not those who trust a marketing chart.

Tagsllama-4long-contextinference-speed

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceLlama 4 inference speed benchmark under rate limitsAnalysis of Llama 4 inference speed rate limits: how API throttling, not model compute, dominates tail latency, and why fallback architectures win.Benchmarks & performanceLlama 4 inference speed: self-hosted vs API providersA head-to-head comparison of Llama 4 self-hosted vs API speed across latency, cost, ergonomics, and limits to help engineers choose the right deployment.Benchmarks & performanceLlama 4 inference speed vs Llama 3.3 70B comparedA head-to-head engineering comparison of Llama 4 vs Llama 3.3 70B speed across latency, cost, capabilities, and ergonomics, with a use-case verdict.Benchmarks & performanceLlama 4 inference speed benchmark on n4n routingPractical analysis of Llama 4 inference speed n4n routing: isolating gateway overhead, provider variance, and cache hints to get real production latency numbers.Benchmarks & performanceLlama 4 Maverick inference speed across nine providersEngineering analysis of Llama 4 Maverick inference speed providers: benchmark method, archetypes, tradeoffs, and how to pick the right one for your workload.Benchmarks & performanceFastest Llama 4 providers ranked by latencyEngineering-ranked list of the fastest Llama 4 providers by latency, with OpenAI-compatible measurement code and fallback routing notes.

All llama 4 inference speed by provider posts →