Radiology departments evaluating AI report assistants rarely benchmark the metric that determines workflow fit: llm latency radiology report generation under realistic hospital load. Treating the task as a single monolithic LLM call hides avoidable delays, and a partitioned pipeline that separates structured drafting from narrative polish consistently delivers faster turnaround without sacrificing clinically relevant accuracy.
The latency trap in end-to-end report generation
Radiology report generation looks like a textbook LLM task: take observations, produce a structured narrative. Most teams implement it as one chat completion with a large model. That choice bakes in avoidable latency because the model spends compute on both mechanical formatting and clinical reasoning in the same forward pass.
A single call to a 70B-class model for a 400-token output can take several seconds of generation time, even when the clinical content is simple. Meanwhile the radiologist is waiting for the Impression section, which is the only part they read immediately. The rest of the report is documentation overhead, not decision-critical.
Shipping a monolithic call also couples your latency floor to the slowest component. If the model briefly degrades or a provider rate-limits you, the entire report stalls. There is no partial result to show.
What actually drives llm latency radiology report generation
Two independent variables dominate: time to first token (TTFT) and token generation throughput. TTFT is governed by prompt processing—the longer the system prompt and few-shot examples, the higher the prefill cost. Throughput is governed by model size and batching.
TTFT vs generation throughput
Frontier APIs often quote TTFT in hundreds of milliseconds, but self-hosted 8B models on a single A100 can hit lower TTFT for short prompts because there is no network round-trip to a multi-tenant cluster. Generation throughput scales inversely with parameter count: an 8B model serves tokens at roughly 2–3x the rate of a 70B model on equivalent hardware.
For radiology, input prompts are templated. You do not need a 32k context window to generate a report from a JSON blob of findings. Trimming the prompt from 1,200 tokens to 300 tokens can cut TTFT by a measurable fraction because the prefill step is linear in input tokens.
Input shape: structured fields beat free dictation
Feeding the model a structured object instead of a transcript reduces ambiguity and output length. The model spends fewer tokens correcting itself.
{
"study": "CT chest w/ contrast",
"findings": {
"lungs": "no nodules",
"mediastinum": "normal",
"pleura": "small left effusion"
}
}Given that input, a constrained decoder or a small model can emit the sections without deliberation. Free dictation requires the model to infer structure, expanding both input and output token counts. If you control the upstream PACS integration, emit structured findings, not audio.
A partitioned pipeline that cuts latency
The thesis: split the job. Use a small local model to draft the Findings and Impression sections from structured input, then use a larger model only to verify consistency and apply institutional phrasing. Stream the Impression first so the radiologist sees it while the rest generates.
Stage 1: local small model drafts sections
An 8B instruct model fine-tuned on report style can produce a passable draft in under a second of generation. Run it with streaming and stop conditions on section headers.
draft_stream = client.chat.completions.create(
model="llama-3.1-8b-instruct",
messages=[{"role":"system","content":"You are a radiology draft assistant. Output only Findings and Impression."},
{"role":"user","content": json.dumps(input_obj)}],
stream=True,
)The small model never sees the full institutional style guide. It only maps fields to sentences.
Stage 2: larger model verifies and polishes
The large model receives the draft plus the original structured findings. Its prompt is short: “Verify the draft matches findings. Rewrite Impression in 2 sentences.” This keeps its output to ~80 tokens, slashing generation time versus authoring from scratch.
Because the large model runs on a shorter output, its slower throughput matters less. You have moved the latency-sensitive path to the fast model.
Streaming the impression first
Because the Impression is clinically urgent, emit it as soon as the small model produces it. The large-model verification can happen asynchronously and flag discrepancies.
def stream_report(input_obj):
for token in draft_stream:
if token.startswith("Impression:"):
yield token # forward immediately to UI as draftThe UI must label it “draft” until verification returns. That is a product requirement, not a nice-to-have.
Benchmark methodology that doesn’t lie
Latency numbers from a single cold run are worthless. You need warm workers, realistic concurrency, and tail percentiles.
Warm vs cold, concurrency, and tail latency
Cold starts from scaled-to-zero endpoints add seconds unrelated to model speed. Run at least 50 warm-up requests. Then measure p50, p90, p99 under concurrency matching your deployment—radiology rooms may fire 5–10 requests per minute at peak.
Using an OpenAI-compatible endpoint that addresses 240+ models and provides automatic fallback lets you run the same benchmark script against multiple backends without client changes, which simplifies comparative benchmarking. When a provider is degraded, the fallback path exercises your resilience logic instead of breaking the test.
Code to measure correctly
Below is a minimal streaming benchmark that captures TTFT and per-token intervals.
import time, openai, statistics
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
def measure(prompt, model):
start = time.perf_counter()
ttft = None
tok_ts = []
stream = client.chat.completions.create(
model=model, messages=[{"role":"user","content":prompt}], stream=True)
for chunk in stream:
now = time.perf_counter()
if ttft is None and chunk.choices[0].delta.content:
ttft = now - start
if chunk.choices[0].delta.content:
tok_ts.append(now)
itls = [ (tok_ts[i]-tok_ts[i-1])*1000 for i in range(1,len(tok_ts)) ]
return ttft*1000, statistics.median(itls) if itls else 0Swap model between an 8B and a 70B to see the throughput gap directly. Run the loop 200 times at concurrency 4 to get p90.
Caching and routing directives
If your system prompt is static (it should be), mark it cacheable. Gateways that honor client routing directives and forward provider cache-control hints can reduce TTFT by reusing prefill across requests. In radiology, where every report uses the same section definitions, this is pure win.
client.chat.completions.create(
model="llama-3.1-8b-instruct",
messages=[{"role":"system","content": SYSTEM_PROMPT,
"cache_control": {"type":"ephemeral"}}],
stream=True,
)The exact cache field varies by provider; the gateway forwards it without translation. Do not put patient data in the cached segment.
Tradeoffs: when the fast path is clinically unsafe
Partitioning is not free. The small model can hallucinate a normal finding from an abnormal input. The verification stage must be mandatory for any report that enters the patient record. If the large model is unavailable, you cannot silently ship the draft.
Regulatory constraints also matter. Local models keep PHI on-prem, but a cloud verification step needs a BAA and possibly de-identification. The latency win from cloud frontier models may be offset by network egress and compliance review.
Another tradeoff: streaming the Impression first can surface a wrong conclusion before verification completes. You must label it “draft” in the UI until the second stage confirms. If the verifier flags a mismatch, the UI must retroactively warn the radiologist who may have already read the line.
Finally, the partitioned pipeline adds orchestration code. You now run two models, manage a draft cache, and handle partial failures. For a team shipping a v0, the monolithic call is simpler. The latency dividend only pays off once volume justifies the operational burden.
Takeaway
Benchmarking llm latency radiology report generation shows that a monolithic call to a large model is the wrong default for production radiology workflows. Partition the pipeline, stream the urgent section, cache the static prompt, and measure p90 under concurrency. Use the small model for structure, the large model for trust. The decisive takeaway: architect for latency before you tune the prompt, because the architecture determines the floor—no prompt trick will recover seconds that a single synchronous call threw away.