32k vs 128k vs 1M: context length latency scaling

Context length latency scaling is the silent tax on every long-context LLM feature. Push a 32k prompt to 128k and you rarely pay 4x the latency; you pay 10x or more in time-to-first-token, and at 1M the system often falls off a cliff into minute-scale waits. This analysis breaks down why the relationship is non-linear, where the bottlenecks actually sit, and how to engineer around them.

The two phases that dominate latency

Every transformer inference call splits into two distinct phases: prefill and decode. Prefill processes the entire input prompt in parallel-ish batches to build the KV cache. Decode generates output tokens one at a time, attending to the cached keys and values.

Latency = time-to-first-token (TTFT, dominated by prefill) + (output_tokens / tokens-per-second). Context length latency scaling is almost entirely a prefill problem, with a smaller secondary drag on decode due to KV cache memory bandwidth.

You can measure TTFT directly against any OpenAI-compatible endpoint:

import time, openai

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
t0 = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": long_prompt}],
    max_tokens=32,
    stream=True,
)
first = None
for chunk in stream:
    if chunk.choices[0].delta.content:
        first = time.perf_counter()
        break
print(f"TTFT: {first - t0:.2f}s for ~{len(long_prompt)//4} tokens")

Run this at 32k, 128k, and 1M and the curve will not be linear.

What actually happens from 32k to 128k

Naive self-attention is O(n²) in sequence length, but production runtimes use FlashAttention or sliced variants that drop the compute to roughly O(n) while remaining memory-bound. The real cost is VRAM and bandwidth. A 70B-class model with grouped-query attention still needs tens of gigabytes of KV cache at 128k tokens. That cache must be allocated contiguously, which fragments the allocator and forces smaller batch sizes.

Smaller batches mean the GPU is underutilized during prefill. A single 128k request can evict an entire batch of 32k requests from the same node. Throughput per watt collapses. So context length latency scaling from 32k to 128k is superlinear not because the math got harder, but because the scheduler lost its ability to hide latency behind concurrency.

Decode drag

Once prefill finishes, each generated token reads the full KV cache. At 128k, that read is a multi-gigabyte memory sweep per token. Tokens-per-second typically drops 20–40% versus a 32k run on the same model, even if TTFT is the headline number.

The 1M context cliff

At 1M tokens you exit the single-GPU regime. The KV cache alone can exceed 200GB for large models, requiring tensor parallelism across many devices. Inter-GPU communication turns prefill into a distributed job. Providers that advertise 1M windows often use sparse attention, chunked offloading, or aggressive eviction—all of which push TTFT from seconds into minutes.

This is not a bug; it is physics. The decisive tradeoff: 1M context is viable for asynchronous document digestion, legal discovery, or overnight codebase indexing. It is not viable for interactive chat where a user waits on the cursor. Engineers who bolt a 1M window onto a latency-sensitive path will ship a broken product.

Mitigations that actually work

You have three levers: shrink the live context, cache the prefill, or move the work off the critical path.

Shrink live context

Retrieval-augmented generation beats stuffing. Embed and fetch the 4k tokens relevant to the query instead of forwarding the 400k repository. Summarization loops—compress the conversation every N turns—keep the working set near 32k indefinitely.

Cache the prefill

Stable prefixes (system prompts, long reference docs) should be marked for provider-side caching. A gateway such as n4n.ai that forwards provider cache-control hints lets you reuse prefill across calls without custom integration:

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {"role": "system", "content": "You are a strict code reviewer.", "cache_control": {"type": "ephemeral"}},
    {"role": "user", "content": "<large stable spec>"}
  ],
  "max_tokens": 1024
}

Subsequent calls with the same prefix skip prefill entirely. The context length latency scaling curve flattens because you only pay the prefill tax once per unique document, not once per question.

Move it off-path

If you must process 1M tokens, queue the job. Return a task ID, poll for completion, notify on finish. The user gets a progress bar instead of a hung socket.

Benchmark methodology for your own stack

Do not trust provider marketing sheets. Measure context length latency scaling on the exact model and region you deploy.

1. Generate synthetic prompts of fixed token counts: 8k, 32k, 128k, 512k, 1M.
2. Keep max_tokens at 32 to isolate TTFT.
3. Run 5 trials each; discard the first (warm cache).
4. Record TTFT and decode TPS.

def padded_prompt(tokens: int) -> str:
    # ~4 chars/token heuristic for English-ish text
    return "word " * (tokens // 2)

for n in [8_000, 32_000, 128_000, 512_000, 1_000_000]:
    prompt = padded_prompt(n)
    # run the timing snippet from earlier, append to results

Plot TTFT vs n on a log-log scale. If the slope exceeds 1.2, your provider is hitting memory walls earlier than claimed.

Decisive takeaway

Default to 32k. It is cheap, fast, and fits most agentic and chat workloads when you trim aggressively. Use 128k only when the task genuinely needs the whole document in one shot, and cache the prefix so repeated queries don’t repay prefill. Treat 1M as a batch processing primitive, never as an interactive context window.

Engineers who internalize context length latency scaling stop throwing tokens at problems and start architecting for the KV cache. That is the difference between a demo and a system that survives contact with production traffic.