Why long-context requests slow down time to first token

The long context time to first token slowdown you observe in production isn’t a single bottleneck but the accumulated cost of prefill, attention memory pressure, and scheduler contention. If you treat TTFT as a black box, you’ll misdiagnose it as “the model is slow” and reach for the wrong fixes.

Prefill is where TTFT is won or lost

The two-phase execution model

Transformers process a request in two distinct phases. Prefill consumes the full input prompt in parallel and builds the key-value (KV) cache. Decode then generates tokens one at a time, attending to the cached states.

Time to first token measures the duration of prefill plus the first decode step. For short prompts, prefill is negligible. For a 100K-token context, prefill can be seconds to tens of seconds depending on hardware and batching.

Why prefill scales with context length

Standard self-attention computes pairwise interactions across all tokens. The naive algorithm is O(n²) in compute and memory for sequence length n. FlashAttention and similar kernels reduce constant factors and memory traffic but do not change the asymptotic growth for dense models.

A 7B model on a single A100 might prefill at ~2000–5000 tokens/sec under ideal conditions; a 70B model halves that or worse. Double the context, roughly double the prefill time, until you hit memory bandwidth or kernel launch limits.

# Rough prefill time estimate (idealized, no batching)
def estimate_prefill_ms(tokens, tokens_per_sec):
    return (tokens / tokens_per_sec) * 1000

print(estimate_prefill_ms(100_000, 3000))  # ~33s for a mid-size model

Those numbers are illustrative; real throughput depends on batch size and fragmentation.

Sources of long context time to first token slowdown

KV cache allocation and fragmentation

Each token requires storing K and V tensors for every layer. At 100K tokens, a 70B model in FP16 needs gigabytes of contiguous-ish device memory. Allocators fragment. If the scheduler can’t find a clean block, it falls back to swapping or recomputation, both of which inflate TTFT.

Prefix caching helps: repeated system prompts or document prefixes map to cached blocks. But cache eviction policies vary. A cache miss forces full prefill.

Scheduler contention and head-of-line blocking

Inference servers batch requests to saturate GPUs. A long-context request enters the queue behind shorter ones. Because prefill is compute-heavy, it can block decode steps for other requests if the scheduler uses simple first-come-first-served. Modern servers use continuous batching, but a 100K prefill still consumes a large slice of the step budget.

This is a major contributor to long context time to first token slowdown in multi-tenant deployments. Your request waits for prior prefills to finish, then pays its own prefill cost.

Attention kernel overhead at extreme lengths

Beyond a certain length, even optimized kernels hit sequence-length limits and must split the input into blocks. Block-wise attention adds launch overhead and reduces arithmetic intensity. Some runtimes cap context size by chaining multiple KV cache segments, which adds bookkeeping latency before the first token emerges.

Measuring the slowdown correctly

Don’t trust dashboard averages. Instrument TTFT per request and break it into queue time, prefill time, and first-decode time. With an OpenAI-compatible client, you can capture it directly:

import time, openai

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
start = time.perf_counter()
stream = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": LONG_PROMPT}],
    stream=True
)
first_token = None
for chunk in stream:
    if chunk.choices[0].delta.content:
        first_token = time.perf_counter()
        break
ttft = first_token - start
print(f"TTFT: {ttft*1000:.0f}ms")

(Replace base_url with your gateway; n4n.ai exposes the same interface and forwards cache-control headers so prefix hits show as lower prefill.)

Log the prompt token count alongside. Plot TTFT vs tokens; the slope is your prefill rate, and the intercept is fixed overhead plus queue delay.

Mitigations that actually work

Prefix caching and explicit cache control

If your prompts share a long system prompt or retrieved document, mark it as a cached prefix. Providers that honor cache_control (or equivalent) skip recomputing those tokens.

{
  "model": "anthropic/claude-3.5-sonnet",
  "messages": [
    {"role": "system", "content": "You are a legal assistant. <10k words of statute>", "cache_control": {"type": "ephemeral"}}
  ]
}

An OpenAI-compatible gateway like n4n.ai forwards provider cache-control hints, so the same request shape works across backends. This directly attacks the long context time to first token slowdown when prefixes are stable.

Chunked prefill

Some servers split prefill into chunks interleaved with decode steps. This reduces head-of-line blocking and keeps p50 TTFT sane, but can increase p99 for very long inputs because the prefill completes over multiple rounds. It’s a tradeoff, not a free lunch.

Choose models with architectural concessions

Sliding window attention (e.g., Mistral-style) bounds per-token cost to a fixed window, turning O(n²) into O(n·w). You lose some global recall but often preserve enough for RAG. Sparse attention and state-space hybrids (Mamba, RWKV) avoid KV cache growth entirely. If your task tolerates approximate global context, these cut TTFT by an order of magnitude.

Tradeoffs you can’t ignore

Prefix caching saves prefill but consumes memory that could serve other tenants; in a shared gateway, your cached blocks may be evicted under pressure. Chunked prefill improves fairness but adds latency for the longest requests. Sliding window models reduce cost but can miss distant dependencies, causing silent quality drops.

There is no configuration that makes 200K-token dense attention as fast as 4K. The long context time to first token slowdown is physics plus scheduling, not a bug to patch.

Takeaway

Measure prefill separately from decode, reuse prefixes aggressively, and match model architecture to context needs. If you must ship dense long-context on a latency budget, precompute and cache the static parts, stream the dynamic query, and accept that TTFT will scale linearly with uncached tokens. Design for the slowdown; don’t wish it away.