Measuring context window latency RAG prompts is not as simple as timing a single API call. Retrieval-augmented generation stacks variable-length context behind a fixed user query, and the resulting latency profile shifts nonlinearly with token count.
Define the latency components
Break the request into stages before writing any code: retrieval, prefill (processing input tokens), and decode (generating output). For context window latency RAG prompts, prefill dominates at large contexts because the model must attend to every input token before emitting the first byte.
Prefill vs decode
Prefill scales with input length; decode scales with output length. Even with FlashAttention, prefill is memory-bandwidth bound and grows faster than linear once the context exceeds GPU SRAM capacity. Measure them separately by streaming and capturing time-to-first-token (TTFT) and inter-token latency.
import time, openai
client = openai.OpenAI(base_url="https://api.example.com/v1", api_key="key")
def timed_call(model, context, query):
start = time.perf_counter()
stream = client.chat.completions.create(
model=model,
messages=[{"role":"system","content":context},
{"role":"user","content":query}],
stream=True
)
ttft = None
tokens = 0
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if ttft is None:
ttft = time.perf_counter() - start
tokens += 1
total = time.perf_counter() - start
return ttft, total, tokensBuild a representative corpus
Synthetic “lorem ipsum” context produces misleading curves because tokenization and attention patterns differ from real documents. Pull actual chunks from your vector store or use a public corpus that matches your domain.
Parameterize by token count
Create discrete buckets: 4k, 8k, 16k, 32k, 64k, 128k. Use a tokenizer to truncate or assemble real retrieved passages to hit the target.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B")
def make_prompt(docs, target_tokens):
ctx = ""
for d in docs:
ctx += d["text"] + "\n"
if len(tok(ctx)["input_ids"]) >= target_tokens:
break
ids = tok(ctx)["input_ids"][:target_tokens]
return tok.decode(ids)Pitfall: padding with repeated text triggers model-specific caching weirdness and skews TTFT. Use diverse documents even if you must repeat sources.
Configure the harness for fair comparison
Fix temperature, max_tokens, and stop sequences. Run each bucket 20+ times to get stable p50 and p95. Randomize order to avoid warm-up bias.
Control provider load
Shared endpoints throttle and batch, distorting numbers. Use a dedicated deployment or spread requests under rate limits. When routing through n4n.ai, the gateway honors client routing directives and forwards provider cache-control hints, so you can pin a model and disable fallback to get clean isolation.
{
"model": "anthropic/claude-3.5-sonnet",
"route": {"fallback": false},
"cache": {"read": true, "write": true}
}Separate retrieval from inference in the SLO
If your SLO is end-to-end, include vector search time. A 200ms Pinecone query is invisible at 30k tokens but dominates at 2k. Log it as a separate span.
Execute the sweep
Run sequentially per bucket to avoid concurrency skew, but parallelize across repetitions with a worker pool capped at your rate limit.
for size in 4096 8192 16384 32768 65536 131072; do
python bench.py --tokens $size --runs 25 >> results.csv
doneWarm vs cold cache
Provider prompt caches cut prefill dramatically if your RAG context is static across calls. Benchmark both: first call cold, subsequent warm. Mark the cache hit in your results.
# pseudo: force cache miss by changing a trailing token
context_cold = context + "\n<!-- run_id=123 -->"
context_warm = context + "\n<!-- run_id=123 -->" # identical on repeatAnalyze the latency curves
Plot TTFT vs context size. You will typically see a knee where linear becomes superlinear due to context swapping or KV-cache fragmentation.
Statistical comparison
Don’t eyeball. Compute confidence intervals.
import numpy as np
def ci(vals, alpha=0.95):
a = np.array(vals)
m = a.mean()
se = a.std(ddof=1) / np.sqrt(len(a))
z = 1.96 if alpha==0.95 else 2.576
return m - z*se, m + z*seIf confidence intervals overlap, treat the difference as noise.
Cost tradeoffs
Longer context burns input tokens even if the model ignores most of it. Per-token metering shows the bill scales with retrieved bytes, not answer quality. Context window latency RAG prompts must be evaluated against both p95 latency and dollar cost per query.
Tradeoff: shrinking context with aggressive reranking lowers latency but risks missing the relevant passage. Measure answer quality with a held-out set alongside latency.
Optimize RAG context packing
After benchmarking, act on the knee point.
Rerank and truncate
Keep top-k passages by relevance score, not all retrieved. Set k from the bucket where TTFT blows up.
def select_topk(docs, scores, k):
paired = sorted(zip(docs, scores), key=lambda x: x[1], reverse=True)
return [d for d,_ in paired[:k]]Compress with summaries
Summarize clusters of chunks before injection. Adds a pre-pass but cuts prefill.
Pitfall: summary drift. Validate that compressed context preserves answer accuracy within 2% on your eval set. Otherwise you traded latency for silent failures.
Use variable max_tokens
Decode phase is pure cost. Set max_tokens to the observed 99th percentile of real answer lengths, not a lazy 1024.
Production monitoring
Benchmark is a snapshot. In production, log TTFT per request tagged by context length bucket.
interface LatencyTag { contextBucket: number; ttftMs: number; model: string }
function record(tag: LatencyTag) {
metrics.histogram('rag.ttft', tag.ttftMs, { bucket: tag.contextBucket })
}Alert when p95 exceeds benchmark baseline by 30%. Degraded providers often show prefill blowup before returning errors. Automatic fallback masks the symptom; keep raw per-provider numbers.
Common pitfalls
- Ignoring retrieval latency: vector search can add 50–200ms; include it in end-to-end.
- Using max_tokens too small: decode truncated, misrepresenting total.
- Not accounting for batching: providers batch requests, hiding true prefill cost.
- Assuming all models scale same: some cap context with sliding window or rope scaling.
- Single-run benchmarks: variance at 128k is high; always report p95 with CI.
Context window latency RAG prompts must be measured against your actual retrieval distribution, not public model cards. The curve you ship is the one you instrument.