Benchmarking LLM latency for real-time clinical notes

llm latency clinical notes is the silent determinant of whether an ambient documentation tool survives contact with a real ward. When a physician finishes dictating, they expect a structured draft before they finish washing hands—not a spinner. Miss that window and the note gets written manually, no matter how clinically accurate the model is.

Why real-time clinical notes are a latency-critical workload

Clinical documentation is not batch processing. The clinician’s cognitive context is live: they are thinking about the patient, the next order, the pager. If the LLM-backed scribe injects a delay longer than the natural pause in conversation, the doctor either waits (bad) or ignores the output (worse).

In ambulatory settings, a typical encounter yields one to five minutes of free speech. The ASR pipeline converts that to text, then the summarization model must produce a SOAP note. The tolerable total latency from end-of-speech to readable draft is roughly two to four seconds in observed clinician demos. That budget must cover network round-trips, inference, and rendering.

What to actually measure

Most latency reports fixate on time-to-first-token (TTFT). For clinical notes, TTFT matters because streaming text on screen reassures the user the system is alive. But the metric that decides adoption is total completion latency at p95 under concurrent load.

Break the LLM portion into three phases:

• Queue and schedule delay: time before the provider accepts the request.
• TTFT: affected by context processing, prefix cache hits, and model size.
• Generation throughput: tokens per second, driving total time for a 300–800 token note.

A note that streams its first token at 400ms but then dribbles at 20 tok/s will feel slower than one that starts at 900ms but sustains 80 tok/s. Measure both.

Inter-token latency and perceived smoothness

A steady stream at 40 tok/s feels better than a bursty 80 tok/s with gaps. Record the standard deviation of inter-token delays. Clinicians read as it streams; a stall mid-sentence triggers doubt about whether the system hung.

Building a faithful benchmark for llm latency clinical notes

Synthetic “write a poem” prompts will lie to you. You need prompts shaped like your production traffic: a system message with rigid instructions, a rolling window of prior encounter notes, and a user message containing the raw transcript.

Example message shape:

[
  {
    "role": "system",
    "content": "You are a clinical scribe. Output a SOAP note. Use concise bullets. Prior encounters: {{prev_notes}}"
  },
  {
    "role": "user",
    "content": "Transcript: Pt states cough x3d, productive yellow sputum, no fever. Exam: lungs clear."
  }
]

Then drive the benchmark with a distribution of input sizes (1k–4k tokens) and output lengths (300–800). Use streaming and record timestamps per chunk.

Minimal Python harness:

import time, openai

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="YOUR_KEY")

def bench(model, messages):
    start = time.perf_counter()
    ttft = None
    stream = client.chat.completions.create(
        model=model, messages=messages, stream=True, temperature=0
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta and ttft is None:
            ttft = time.perf_counter() - start
    total = time.perf_counter() - start
    return ttft, total

Concurrency profile

Simulate multiple exam rooms finishing notes simultaneously at the end of a clinic block. Use asyncio or threads to fire 8–16 parallel generations. A model that looks fine at n=1 can fall apart at n=12 because of provider queueing.

import asyncio, time, openai
client = openai.AsyncOpenAI()

async def bench_async(model, messages):
    start = time.perf_counter()
    ttft = None
    stream = await client.chat.completions.create(
        model=model, messages=messages, stream=True, temperature=0
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content and ttft is None:
            ttft = time.perf_counter() - start
    return ttft, time.perf_counter() - start

# asyncio.gather(*[bench_async(m, msgs) for _ in range(12)])

Run each candidate model with at least 50 samples at matched concurrency. Report p50 and p95, not averages.

Model tiering: small models for draft, large for edge cases

The reflex is to pick the smartest model. That’s wrong for real-time clinical notes. A frontier model may produce a cleaner note but its TTFT and generation time under load can blow the three-second budget.

We recommend a tiered approach:

• Tier 1 (default): A small instruction-tuned model (e.g., GPT-4o-mini, Claude Haiku, Llama-3.1-8B) for routine visits.
• Tier 2 (fallback/complex): A larger model triggered only when confidence is low (ambiguous transcript, rare disease lexicon).

The tradeoff is post-edit burden. Small models hallucinate less than they used to, but they may drop a negative finding. You must measure clinical error rate alongside latency—not as a separate project, but as a joint objective. If a small model saves 1.5s but adds 30 seconds of clinician correction, it loses.

Cache prefixes to kill TTFT

Clinical prompts are repetitive. The system instruction and the prior notes block change slowly within a session. Providers like Anthropic and OpenAI support prompt caching; an OpenAI-compatible gateway such as n4n.ai forwards provider cache-control hints so the cached prefix is honored across requests. This turns a 3k-token context load into a cached read, often cutting TTFT by half.

Implement it by marking the static portion:

client.chat.completions.create(
    model="claude-3-haiku",
    messages=messages,
    extra_headers={"anthropic-cache-control": "ephemeral"}
)

If your gateway normalizes this, you avoid per-provider SDK forks. The latency win is real, but watch cache eviction: a quiet clinic at 2am may have cold caches, so benchmark at both warm and cold states.

Routing and fallback to tame the tail

The worst latency killer is a hung provider. A single stuck request at p99 destroys trust. Client-side timeout and fallback is mandatory:

def bench_with_fallback(messages, models, timeout=2.0):
    for m in models:
        try:
            return bench_with_timeout(m, messages, timeout)
        except TimeoutError:
            continue
    raise RuntimeError("all models failed")

An inference gateway that performs automatic fallback when a provider is rate-limited or degraded removes this boilerplate and protects your p95. But you still pay the serialization and network cost of the retry, so set aggressive client timeouts.

Compliance boundaries shift the numbers

If you process PHI, you likely route through a HIPAA-eligible endpoint or a self-hosted vLLM instance in an isolated VPC. Latency there is not the same as the public API. A self-hosted T4 GPU will beat a frontier model on TTFT but lose badly on throughput for long outputs. Benchmark the exact deployment topology you will ship, not the vendor’s marketing endpoint.

Decisive takeaway

Stop quoting provider “average latency” sheets. For llm latency clinical notes, build a benchmark that mirrors your prompt shape, turns on streaming, and records p95 total completion under realistic concurrency. Use a small model as the default scribe with prefix caching, escalate to a larger model only on detected complexity, and enforce client-side timeouts with fallback. If a model cannot return a complete SOAP draft within your clinician’s tolerance (we treat 2–3s p95 as the line) in your own VPC, it is not viable regardless of its MMLU score. Ship the tier, measure the errors, and iterate.