How prompt length affects time-to-first-token

Prompt length is the single biggest driver of time-to-first-token (TTFT) in LLM inference. Every token in your prompt must be processed during the prefill phase before the model can emit a single output token, and that prefill cost scales superlinearly with sequence length on most hardware. If you’re optimizing for latency, controlling prompt size isn’t optional — it’s the primary lever.

The prefill bottleneck

LLM inference splits into two distinct phases: prefill (processing the prompt) and decode (generating tokens autoregressively). TTFT is almost entirely determined by prefill latency. During prefill, the model computes attention over the entire prompt sequence simultaneously, populating the KV cache that decode will reuse.

The computational complexity of standard self-attention is O(n²) in sequence length n. FlashAttention and kernel fusion reduce the constant factors dramatically, but the asymptotic scaling remains: doubling prompt length roughly quadruples prefill FLOPs. On H100s with FP8, a 4K prompt might take 15 ms prefill; 16K pushes past 100 ms. The relationship holds across model sizes — larger models just have higher absolute numbers.

# Rough prefill latency model for a dense transformer
# Assumes flash attention, batch size = 1
def estimate_prefill_ms(prompt_tokens: int, model_dim: int, n_layers: int, gpu_tflops: float) -> float:
    # 2 * n² * d * L FLOPs for attention (QK^T + AV), plus MLP ~ 8 * n * d² * L
    attn_flops = 2 * prompt_tokens**2 * model_dim * n_layers
    mlp_flops = 8 * prompt_tokens * model_dim**2 * n_layers
    total_flops = attn_flops + mlp_flops
    return (total_flops / (gpu_tflops * 1e12)) * 1000  # ms

# Llama-3-70B-ish: dim=8192, layers=80, H100 FP8 ~2000 TFLOPs
for n in [1024, 4096, 16384, 32768]:
    print(f"{n:5d} tokens -> {estimate_prefill_ms(n, 8192, 80, 2000):.1f} ms")

Output:

 1024 tokens ->    8.2 ms
 4096 tokens ->  112.4 ms
16384 tokens -> 1750.2 ms
32768 tokens -> 6944.3 ms

Real-world numbers include kernel launch overhead, memory bandwidth limits, and scheduler effects, but the quadratic trend dominates. This is why TTFT jumps from “imperceptible” to “user-visible” right around 4–8K tokens on current hardware.

Why it’s not purely quadratic

Three factors complicate the simple O(n²) story:

1. Memory bandwidth saturation. For short prompts (< 2K), compute units sit idle waiting for data. The roof-line model shifts from compute-bound to memory-bound. You’ll see sub-quadratic scaling at the low end because the GPU isn’t fully utilized.

2. Chunked prefill / prefix caching. Some serving engines (vLLM, TGI, TensorRT-LLM) split long prefill into chunks to interleave with decode batches. This reduces tail latency for other requests but adds scheduling overhead. Prefix caching — reusing KV cache for shared prompt prefixes — can make repeated long prompts nearly free on subsequent calls, but cold-start TTFT still pays the full cost.

3. Model architecture variations. Models with grouped-query attention (GQA) or multi-query attention (MQA) reduce the KV cache size and attention FLOPs proportionally to the head ratio. A 70B model with 8 KV heads (vs 64 Q heads) cuts attention cost ~8x versus MHA. Sliding window attention (Mistral, Gemma 2) caps effective context for attention computation, making prefill scale linearly beyond the window size — but you still pay full cost up to the window.

Batch size interacts with prompt length

In production, you rarely run batch size = 1. Continuous batching packs multiple requests into a single forward pass. The prefill for a new request gets concatenated with decode steps for in-flight requests. This creates contention:

Long prefill monopolizes SMs and memory bandwidth, stalling decode for other requests
Short requests stuck behind a long prefill see inflated TTFT
The scheduler’s chunking policy (max prefill chunk size, preemption granularity) becomes a latency knob

# vLLM-style chunked prefill config
# Smaller chunks = better fairness, worse throughput
chunked_prefill_config = {
    "max_num_batched_tokens": 8192,      # total tokens per forward pass
    "max_num_seqs": 256,                 # max sequences in batch
    "prefill_chunk_size": 512,           # tokens per prefill chunk
    "enable_prefix_caching": True,
}

If you route a 32K prompt into a server tuned for 4K typical prompts, you’ll starve everyone else. This is why request-level routing directives matter — sending long-context requests to dedicated instances or model variants (e.g., a 128K-capable model with sliding window) protects the fast path for short prompts.

Prompt compression: what actually works

Engineers reach for prompt compression when TTFT budgets are tight. Three approaches with different tradeoffs:

1. Semantic compression (LLMLingua, LongLLMLingua)

Train a small model to identify and remove low-information tokens. Typical claims: 2–4x compression with < 1% quality drop on QA tasks. Reality: works well for retriever-heavy RAG prompts where context contains lots of boilerplate. Fails on code, structured data, or prompts where every token carries signal. Adds its own inference latency (though the compressor is small).

2. KV cache quantization / eviction

Quantize KV cache to INT4/INT8 (cuts memory bandwidth, speeds prefill slightly). Evict “unimportant” KV positions during prefill based on attention scores (H₂O, SnapKV). Risk: quality degradation on tasks needing full context (needle-in-haystack, long-code reasoning). Most effective when you know the task tolerates lossy context.

3. Prompt restructuring (structural, not learned)

Move static instructions to system prompt → enables prefix caching
Deduplicate few-shot examples → use a single example + “follow this pattern”
Replace verbose formatting (XML, JSON schemas) with compact delimiters
Offload reference data to RAG retrieval at decode time (tool use) rather than stuffing into prompt

Structural compression is free, deterministic, and often yields 30–50% token reduction with zero quality loss. Do this first.

The decode phase doesn’t care (much)

Once prefill finishes, decode latency per token is roughly constant with respect to prompt length — it’s O(1) per step (one matrix-vector multiply per layer, plus attention over cached KV). The KV cache grows linearly, so memory bandwidth per decode step increases slightly, but on modern kernels this is negligible until cache exceeds GPU memory and spills to CPU/NVMe.

This asymmetry — prefill scales with prompt, decode doesn’t — means TTFT and throughput have different optimization profiles. You can optimize TTFT by shrinking prompts, chunking prefill, or using smaller models for the first hop. You optimize throughput by maximizing batch size, KV cache quantization, and speculative decoding. They pull in opposite directions.

Practical rules of thumb

Prompt length	Expected TTFT (H100, 70B, FP8)	Dominant bottleneck	Mitigation
< 512	2–5 ms	Kernel launch, scheduler	Batch aggressively
512 – 2K	5–15 ms	Memory bandwidth	Prefix caching, merge short requests
2K – 8K	15–80 ms	Compute (attention)	Chunked prefill, GQA models
8K – 32K	80–400 ms	Compute + memory	Dedicated long-context workers, sliding window models
> 32K	400 ms – 2 s+	KV cache size, OOM risk	RAG / tool use instead of stuffing, model with 128K+ native context

These numbers assume cold start, no prefix cache hit, batch size 1. Continuous batching with mixed lengths will show higher variance.

Routing as a latency control plane

Since prompt length is often determined upstream (user input, retrieved docs, conversation history), the inference layer needs to react. A gateway that inspects prompt_token_count before routing can:

Send short prompts to a low-latency pool (smaller model, higher batch density, speculative decoding enabled)
Send long prompts to a high-memory pool (larger model, chunked prefill, prefix caching warm)
Reject or truncate prompts exceeding a hard TTFT SLA
Strip cache-control hints from providers that expose them (e.g., x-cache-status: hit on repeated prefixes)

This is where n4n.ai’s routing directives come into play — the client can tag requests with x-n4n-max-ttft-ms or x-n4n-prefer-latency, and the gateway selects the appropriate backend pool automatically. The provider’s own cache-control headers flow back through, so the caller knows whether prefix caching actually fired.

The decisive takeaway

Prompt length is the TTFT budget. Every token you add to the prompt buys you latency — quadratically at the low end, linearly at the high end, but always monotonically. The only sustainable strategies:

Measure it. Log prompt_tokens and ttft_ms per request. Plot the curve for your model + hardware. It will not be a straight line.
Compress structurally first. Deduplicate, offload to tools, enable prefix caching. This is free latency reduction.
Route by length. Don’t let a 50K token request share a worker with a 200 token chat turn. The short request will pay the long request’s prefill tax.
Accept the physics. If your product requires 100K context and sub-100ms TTFT, you need either a different model architecture (linear attention, SSM, hybrid) or a different product design (progressive loading, streaming retrieval). No amount of kernel tuning fixes O(n²) prefill on transformer attention.

The engineers who ship low-latency LLM products treat prompt tokens like a currency they spend carefully. The ones who don’t, ship dashboards showing “P99 TTFT: 12 seconds” and wonder why users churn.