A 100k token prompt response time is dominated by prefill, not token generation, and the penalty is paid upfront as time-to-first-token. If you treat large prompts as a simple multiplier on latency, you will misdesign your system. The real cost depends on whether the prefix is cached, the serving stack’s attention implementation, and batching dynamics.
The anatomy of latency in LLM inference
Every LLM completion splits into two phases. Prefill processes the input tokens and builds the key-value (KV) cache. Decode generates output tokens one (or a few) at a time, attending to the KV cache.
Prefill vs decode
Prefill is compute-bound and parallel: the model runs the forward pass over all input tokens simultaneously. Decode is memory-bandwidth-bound and sequential: each step fetches the full KV cache and computes a single token.
A 1k token prompt might prefill in 20–50 ms on a well-optimized serving stack. A 100k token prompt is two orders of magnitude larger. Even with linear-time fused attention, you are moving and computing on 100× the tensors. The 100k token prompt response time therefore shows up as a multi-second TTFT unless the prefix is already cached.
Decode latency is independent of prompt size beyond the cost of attending to a larger KV cache. That cost grows sublinearly with sequence length on modern GPUs because attention is memory-bound, not compute-bound, at long contexts.
Where the 100k tokens hit
The user perceives three numbers:
- TTFT (time to first token): includes prefill + scheduling + network.
- TPS (tokens per second during streaming): decode speed.
- Total latency: TTFT + (output_tokens / TPS).
Doubling output length doubles total latency. Doubling input length from 1k to 100k can increase TTFT by 50–100× if uncached. That asymmetry is the core finding.
Measuring 100k token prompt response time
You cannot reason about this without measuring. Here is a minimal streaming probe against any OpenAI-compatible endpoint:
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")
# Build a ~100k token prompt (e.g., repeat a known string; real docs vary)
big_prompt = "The quick brown fox. " * 5000 # ~30k tokens; scale as needed
start = time.perf_counter()
stream = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": big_prompt}],
stream=True,
stream_options={"include_usage": True},
)
ttft = None
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
ttft = time.perf_counter() - start
break
print(f"TTFT: {ttft:.2f}s")Run this with and without a warm prefix cache. The delta is your true 100k token prompt response time tax. Do not trust provider dashboard averages—they mix batched and isolated requests.
Why the relationship isn’t linear
Naive self-attention is O(n²) in sequence length. FlashAttention and variants reduce the asymptotic cost and memory footprint, making prefill roughly linear in practice. But constants matter:
- Batch contention: A 100k token request occupies a large slice of GPU memory for its KV cache. Schedulers like vLLM or TensorRT-LLM will delay or chunk it behind smaller requests. Your TTFT suffers from queueing, not just compute.
- Quantization: FP8 or INT4 KV caches shrink memory traffic, improving prefill throughput but not eliminating the linear term.
- Model architecture: MoE models (e.g., Mixtral) keep active params low, so prefill scales with routed experts, not total param count. Dense models pay full cost.
A common mistake is to extrapolate from a 4k token test. At 100k, you hit memory fragmentation and eviction policies. The 100k token prompt response time can degrade non-monotonically as the scheduler sheds other workloads.
The caching escape hatch
If the first 100k tokens are identical across requests (system prompt, codebase, document), prefix caching turns the tax into a rounding error. Providers expose this differently:
- Anthropic:
cache_control: { "type": "ephemeral" }on a prompt block. - OpenAI: implicit prompt caching for repeated exact prefixes.
- Gemini: cached content objects.
An inference gateway such as n4n.ai honors client routing directives and forwards provider cache-control hints, so you can reuse a 100k prefix across requests without rewriting your client for each backend.
Example curl with cache hint passed through:
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-3-5-sonnet",
"messages": [
{"role": "system", "content": "<<100k tokens>>", "cache_control": {"type": "ephemeral"}},
{"role": "user", "content": "Summarize section 4."}
]
}'When the cache hits, TTFT drops from seconds to tens of milliseconds. The 100k token prompt response time becomes a non-issue—provided your prefixes are stable and you respect minimum cacheable lengths (often 1024+ tokens).
Real-world tradeoffs: when to send the whole document
Full context vs RAG
Sending 100k tokens per query is simplest: no retrieval pipeline, no chunking, no relevance scoring. But you pay the prefill tax on every uncached call. For a latency-sensitive chat UI, a 3-second TTFT is unacceptable.
Retrieval-augmented generation cuts prompt size to relevant 2–8k tokens. You trade system complexity for predictable sub-second TTFT. For asynchronous jobs (document analysis, batch summarization), the full-context approach is fine because users tolerate minutes.
Compression and summarization
You can compress the prefix with a cheaper model first:
# Pseudocode: pre-summarize long doc with a fast model
summary = cheap_client.chat.completions.create(
model="mistral-7b",
messages=[{"role": "user", "content": f"Compress:\n{raw_doc}"}],
)
final = premium_client.chat.completions.create(
messages=[{"role": "user", "content": f"{summary} Answer: {question}"}]
)This trades an extra call for a 10× smaller prompt on the expensive model. The math works when the cheap model’s latency is less than the prefill savings.
Streaming and perceived latency
If you must send 100k tokens, stream immediately and show a spinner. Users forgive 2-second TTFT if tokens arrive smoothly afterward. But if your stack buffers the whole completion, the 100k token prompt response time compounds with generation, and timeouts loom.
Hardware and serving reality
Current H100/B200 deployments achieve prefill throughput in the thousands of tokens per second per GPU for batched traffic, but a single isolated 100k request may saturate a device’s KV cache capacity. Multi-GPU tensor parallelism helps, but communication overhead grows with context length.
Continuous batching masks some prefill cost by interleaving decode steps of other requests. However, a 100k prompt still allocates a massive KV block; if the scheduler is full, your request waits. This is why identical-load benchmarks lie: production mixes short and long inputs.
Decisive takeaway
Measure TTFT and decode separately; never report a single “latency” number for long-context calls. A 100k token prompt response time is a prefill problem, not a generation problem. If your prefix is static, use provider prefix caching and the tax vanishes. If it changes per request, cap prompt size with RAG or summarization for interactive paths, and reserve full 100k contexts for offline or batch workloads. Build your client to tag cache boundaries and route long prompts to backends that support them—and your users will never feel the 100k tokens.