Reporting average latency vs percentile latency for an LLM inference path is like describing a commute by its mean travel time: it looks fine until the bridge is up. The mean smooths over the requests that time out, queue behind a stalled GPU, or hit a provider’s cold path. If you operate a gateway or call models in production, the only latency number that predicts user complaints is the tail.
The averaging trap
Why means collapse the signal
A mean is a sum divided by a count. It is mathematically blind to order and shape. Take 99 requests that finish in 200 ms and one request that finishes in 20,000 ms. The average is roughly 397 ms. That single slow request is invisible to anyone scanning a dashboard of averages, yet it is the request that triggered a client timeout, a retry storm, or a user abandoning the session.
Latency is not a single number you optimize; it is a distribution you manage. The average latency vs percentile latency comparison matters because the percentile preserves the shape. The p99 tells you the worst 1% experience. The p50 tells you the typical experience. The mean tells you neither with fidelity.
LLM inference is not a Gaussian process
Classic network services sometimes approximate a bell curve under stable load. LLM inference does not. The serving path includes:
- Queue time behind a batch scheduler.
- Time to first token (TTFT) dominated by prompt processing and KV-cache availability.
- Generation time proportional to output length, which is itself model-generated and unbounded.
- Provider-side rate limits that inject explicit 429 delays.
- Cold starts on less-used model variants or regions.
These modes produce a mixture distribution: a tight cluster of fast successes, a long tail of stalled or retried calls, and occasional wall-clock spikes when a upstream provider degrades. Averaging across modes yields a number that matches no actual request.
What percentile latency actually tells you
Definitions that matter
- p50 (median): Half of requests are faster, half slower. This is your baseline user expectation.
- p95: The slowest 5% boundary. Useful for spotting intermittent degradation.
- p99: The slowest 1% boundary. This is where SLOs break and where fallback logic earns its keep.
- p999 (if you run high volume): Catastrophic tail, often provider-specific.
When you plot these, the gap between p50 and p99 is the latency spread. A healthy gateway shows p99 under 3x p50 for interactive models. A sick one shows p99 at 20x or more.
Tail latency is user-visible latency
In a chat UI, the user waits for the first token. If your p99 TTFT is 8 seconds but your average is 400 ms, one in a hundred messages looks broken. In agentic loops where a planner calls a model ten times, the compounded tail probability is no longer 1% per call—it is roughly 10% that at least one call hits the tail. Average latency vs percentile latency analysis makes this compounding obvious only when you look at the distribution.
Measuring it correctly
You cannot manage a percentile you do not record. Emit raw request durations, not just pre-aggregated averages.
Compute from raw samples
import numpy as np
# durations in milliseconds from your access log or OTel exporter
durations_ms = [210, 230, 225, 240, 199, 5200, 215, 222, 235, 250, 180, 260]
p50 = np.percentile(durations_ms, 50)
p95 = np.percentile(durations_ms, 95)
p99 = np.percentile(durations_ms, 99)
print(f"p50={p50:.0f}ms p95={p95:.0f}ms p99={p99:.0f}ms")If you cannot ship NumPy to a sidecar, sort and index:
def percentile(sorted_vals, p):
if not sorted_vals:
return None
k = int(len(sorted_vals) * p)
k = min(k, len(sorted_vals) - 1)
return sorted_vals[k]
sorted_d = sorted(durations_ms)
print(percentile(sorted_d, 0.99))Export a distribution, not a point
A metrics payload should carry the shape:
{
"metric": "llm_request_duration_ms",
"route": "gpt-class",
"p50": 235,
"p95": 880,
"p99": 5100,
"sample_count": 14820,
"window": "5m"
}Instrument the right sub-metrics
For streaming LLMs, split the latency into phases:
ttft_ms: from request received to first token emitted.tpot_ms: average inter-token gap during generation.total_ms: through final token.
The tail on TTFT is usually where providers fail. The tail on TPOT shows GPU saturation. Averaging them together hides which lever to pull.
A quick log scan with standard tools:
# assume latency_log.tsv has duration in last column
awk -F'\t' '{print $NF}' latency_log.tsv | sort -n | \
awk 'BEGIN{c=0} {a[++c]=$1} END{print "p99=" a[int(c*0.99)] "ms"}'Tradeoffs of chasing tails
The cost of p99 optimization
Driving p99 down often conflicts with driving mean down. Batching requests improves throughput and average latency by sharing GPU compute. But a batch head-of-line block means one slow prompt delays others, pushing p99 up. To protect the tail you provision spare capacity, limit batch size, or shed load earlier. That costs money.
Automatic fallback is another tail tool with a tail cost. Routing a timed-out call to a second provider adds the retry latency to that request. If your fallback is unconditioned, you can turn a 20 s stall into a 25 s stall for the user. You must measure p99 per route to confirm the fallback helps.
When average latency still matters
Average latency vs percentile latency is not a religion. For offline batch jobs—bulk embeddings, nightly document summarization—throughput per dollar dominates. There, the mean token cost and mean request time per thousand items predict your bill. You still want to know the max, but a p99 spike on a batch job that runs at 2 a.m. is tolerable if it does not block the pipeline.
The decisive split: interactive paths need tail SLOs; batch paths need mean efficiency.
Gateway-specific concerns
An OpenAI-compatible gateway that fronts 240+ models and performs automatic fallback when a provider is rate-limited must monitor p99 on each route. Average latency vs percentile latency reporting would mask a degraded provider that only slows 5% of calls but triggers fallback storms. n4n.ai honors client routing directives and forwards provider cache-control hints, so per-route p99 lets you verify that a cache hit or fallback actually reduced worst-case latency rather than just nudging the mean.
When you add fallback, graph p99 before and after. If the post-fallback p99 is lower than the primary-only p99, the feature works. If it is higher, your fallback threshold is too loose.
Also watch the interaction with cache-control. A provider cache hit can drop TTFT from seconds to milliseconds; a missed cache on a long prompt sits in the tail. Forwarding cache-control hints from the client to the provider is only useful if you can attribute p99 improvements to cache hits versus misses. Tag your metrics accordingly.
Decisive takeaway
Stop leading with average latency. Record raw durations, export p50/p95/p99 per route and per phase, and set your SLO on p99 for any interactive model call. Use the mean only for cost accounting on batch workloads. The gap between your p50 and p99 is the size of the problem you have not yet seen; close it with targeted fallback, capacity, and cache hints, then prove the fix with the tail metric—not the average.