Benchmarking llm latency clinical decision support requires a methodology built around clinician workflows, not the synthetic hello-world prompts used in generic LLM dashboards. In a triage assistant or drug-interaction checker, the cost of a slow response is not an abandoned conversation but a clinician skipping the tool entirely and relying on memory.
Why clinical latency is not chatbot latency
A consumer chatbot tolerates a three-second pause; the user is already reading. A clinical decision support (CDS) tool sits inside an ordering screen or a note template. The clinician expects an answer before they tab away. If the model streams slowly, they assume the integration is broken. That makes tail latency, not average latency, the only number that matters for adoption.
We have watched teams ship a perfectly accurate model that failed because p99 time-to-first-token exceeded four seconds. The median was 800 ms. The clinicians never saw the median.
Metrics that matter: TTFT, TPS, and tail
Break latency into two independent variables:
- Time to first token (TTFT): perception of responsiveness.
- Tokens per second (TPS): generation throughput after the first token.
- Total latency: TTFT + (output_tokens / TPS).
For llm latency clinical decision support, TTFT drives trust; TPS drives how long the clinician waits for the full suggestion. Measure both separately. Aggregate with p50, p95, and p99. Never report only the mean.
A 200-token completion at 50 TPS takes 4 seconds of generation. If TTFT is 1.2 s, total is 5.2 s. A faster model with TTFT 300 ms but 30 TPS yields 1.2 + 6.7 = 7.9 s. The first is better for short answers, the second may be acceptable if the clinician reads while it streams. You cannot infer this from a single number.
Designing a representative benchmark
Synthetic prompts like “Summarize a clinical note” hide the real distribution. Pull 50–100 de-identified examples from your own logs: triage questions, lab interpretation, medication reconciliation. Keep the input length distribution realistic—EHR contexts often hit 2k–8k tokens before any output.
Run each example multiple times against the same model and provider to capture cold-start and scheduling variance. Use a minimum of 200 samples per model to get a stable p99. Fewer than that and your tail estimate is noise.
Implementing the harness
Below is a minimal streaming benchmark. It records TTFT and total, then computes percentiles offline.
import openai, time, statistics, json
client = openai.OpenAI() # configure base_url to your gateway
PROMPTS = [
"54M chest pain radiating to left arm, ST elevation. Immediate interventions?",
"Drug interaction between warfarin and fluconazole, management?",
# ... load from file in practice
]
def measure(prompt, model):
start = time.perf_counter()
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True,
)
ttft = None
tokens = 0
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if ttft is None:
ttft = time.perf_counter() - start
tokens += 1
total = time.perf_counter() - start
return ttft * 1000, total * 1000, tokens
results = []
for p in PROMPTS:
for _ in range(10): # 10 repeats × N prompts
results.append(measure(p, "gpt-4o-mini"))
ttfts = sorted(r[0] for r in results)
totals = sorted(r[1] for r in results)
def pct(xs, p):
return xs[int(len(xs) * p)]
print("TTFT p50/p95/p99:", pct(ttfts, .5), pct(ttfts, .95), pct(ttfts, .99))
print("Total p50/p95/p99:", pct(totals, .5), pct(totals, .95), pct(totals, .99))Run this against each candidate model. If you use a gateway that honors client routing directives and forwards provider cache-control hints, send the same cache_control headers you ship in production—prefix caching can cut TTFT markedly on long contexts.
Provider variance and fallback behavior
Model name is not a latency guarantee. The same gpt-4o-mini request routed to Azure, OpenAI direct, or a reseller will show different TTFT distributions. In our measurements across providers, p99 TTFT spread is often 2–3× between the best and worst endpoint on the same model.
This is where an inference gateway changes the benchmark. n4n.ai provides one OpenAI-compatible endpoint addressing 240+ models and performs automatic fallback when a provider is rate-limited or degraded. If your harness hits a degraded provider, the gateway reroutes transparently. Your p99 then reflects the fallback path, not a single provider’s bad minute. That is the number you should design SLOs around, because production will see the same fallback.
Build a chaos step into benchmarking: artificially throttle or block the primary provider and confirm the fallback model meets your TTFT ceiling. If it does not, you have a model-selection problem, not a routing problem.
Model selection tradeoffs
Smaller models (e.g., 8B–70B class) give lower TTFT and higher TPS but may miss nuanced interactions. Larger frontier models improve accuracy at the cost of latency and price. For llm latency clinical decision support, a two-tier strategy works: route simple checks (dose rounding, allergy flags) to a fast model; escalate ambiguous cases to a larger model via a classifier. The classifier itself must be sub-100 ms, or you eat the savings.
Do not assume quantization helps uniformly. Int4 variants reduce memory footprint but can degrade TTFT on short prompts due to kernel overhead. Benchmark the quantized build explicitly; never trust the vendor’s latency claim.
Setting SLOs and the decisive takeaway
Define a hard TTFT SLO (e.g., 1.5 s p95) and a total-latency SLO based on output length. Alert when p99 breaches these for two consecutive measurement windows.
The thesis is simple: benchmark llm latency clinical decision support as a tail-sensitive, streaming-aware system using real prompts and provider fallback, or you will ship a tool clinicians silently ignore. Measure TTFT and TPS separately, sample at least 200 times per model, inject provider failure, and pick models by p99, not by marketing sheets. Accuracy without responsiveness is a shelfware model.
If you route through a gateway, mirror production routing in the harness. The only valid latency number is the one the clinician actually experiences.