A credible speech-to-speech latency benchmark separates signal from orchestration noise. Most published reports lump end-to-end wall-clock time without breaking down automatic speech recognition (ASR), language model inference, and text-to-speech (TTS), which hides where the milliseconds actually go. This analysis argues that you must instrument each stage independently and simulate provider degradation to trust any speech-to-speech latency benchmark.
Why end-to-end numbers lie
A stopwatch measurement from microphone capture to speaker playback is what users feel, but it is useless for engineering. If your voice agent reports 1.8 seconds median latency, you cannot tell whether Whisper queued behind a batch job, the LLM waited for a cold container, or the TTS vendor throttled you.
Worse, aggregated averages mask the tail. A stack that averages 900 ms but p99s at 4 s will feel broken in production. The only way to optimize is to put timestamps at every boundary: after ASR final transcript, at first LLM token, at last LLM token, and at first audio frame out of TTS.
The pipeline stages and what to measure
ASR
Streaming ASR returns partial hypotheses within 200–400 ms on capable cloud services, but the final transcript often lands 400–800 ms after speech ends. Local models like Whisper tiny on CPU can process a 2-second clip in ~300 ms but sacrifice accuracy on noisy input. Measure both partial latency and final-confidence latency.
LLM turn
Time-to-first-token (TTFT) dominates perceived responsiveness. A non-streaming call that generates 30 tokens at 50 ms/token adds 1.5 s of dead air. Streaming flips this: you can start TTS as soon as the first phrase completes. Record TTFT separately from total generation time.
TTS
Neural TTS introduces its own floor. Cloud endpoints (Azure, OpenAI tts-1) typically synthesize a short sentence in 200–400 ms. Local GPU synthesis with Coqui or Piper can hit 100–150 ms. Crucially, measure time-to-first-audio-byte, not just full clip duration, if you support incremental playback.
Network and orchestration
TLS handshake, JSON serialization, and any gateway adds fixed overhead. A direct localhost call avoids this; a routed call through a proxy adds 5–20 ms per hop. Include these in the stage breakdown or you will misattribute latency to the models.
Building a minimal instrumented harness
Below is a Python snippet using the official OpenAI SDK against an OpenAI-compatible endpoint. It timestamps each stage so you can compute a real speech-to-speech latency benchmark instead of a black-box number.
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")
def stage_benchmark(audio_path: str) -> dict:
t0 = time.perf_counter()
with open(audio_path, "rb") as f:
transcript = client.audio.transcriptions.create(model="whisper-1", file=f)
t_asr = time.perf_counter()
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": transcript.text}],
stream=True,
)
first_token_t = None
full_text = ""
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if first_token_t is None:
first_token_t = time.perf_counter()
full_text += delta
t_llm_done = time.perf_counter()
with client.audio.speech.with_streaming_response.create(
model="tts-1", voice="alloy", input=full_text
) as resp:
_ = resp.read()
t_tts = time.perf_counter()
return {
"asr_ms": (t_asr - t0) * 1000,
"ttft_ms": (first_token_t - t_asr) * 1000,
"llm_total_ms": (t_llm_done - t_asr) * 1000,
"tts_ms": (t_tts - t_llm_done) * 1000,
"e2e_ms": (t_tts - t0) * 1000,
}Run this against representative audio samples and network conditions. Store p50, p95, and p99 for each key.
Cloud versus local stacks
A fully local stack (Whisper + llama.cpp + Piper) removes network variance and can achieve sub-second end-to-end on a decent GPU. The tradeoff is model quality: small local LLMs hallucinate more on ambiguous voice commands, and local TTS lacks the prosody of commercial clouds.
A fully cloud stack buys accuracy but introduces dependency on someone else’s capacity. During peak hours, ASR and TTS endpoints degrade. If you call them serially, a 300 ms ASR spike cascades into TTS scheduling delay.
A hybrid approach routes LLM calls through an inference gateway while keeping ASR/TTS local, or vice versa. This is where orchestration policy matters more than raw model speed.
The fallback tail latency problem
Providers get rate-limited. When a primary LLM returns 429, your client either blocks or fails over. If you front your model calls with an OpenAI-compatible gateway such as n4n.ai that honors client routing directives and automatically falls back when a provider is rate-limited, your p99 may improve—but only if the fallback model is warm and the gateway forwards cache-control hints correctly. A speech-to-speech latency benchmark that ignores failover paths will report optimistic tails that never survive contact with production traffic.
To measure this honestly, inject fault tests: force a 503 from the primary, then time the full stage breakdown including fallback connection establishment. The delta between happy-path TTFT and degraded TTFT is your true tail risk.
Streaming synthesis and incremental playback
The biggest perceived-latency win is overlapping stages. Instead of waiting for full LLM text, buffer sentence boundaries and push them to TTS chunk by chunk:
import re
sentence_split = re.compile(r"(?<=[.!?])\s+")
buffer = ""
for chunk in stream:
buffer += chunk.choices[0].delta.content or ""
sentences = sentence_split.split(buffer)
if len(sentences) > 1:
for sent in sentences[:-1]:
synthesize_and_play(sent) # fire-and-forget to audio queue
buffer = sentences[-1]
if buffer:
synthesize_and_play(buffer)This turns a serial pipeline into a parallel one. Your effective latency becomes max(ASR_final, TTFT + first_sentence_TTS) rather than sum of all stages.
What an honest speech-to-speech latency benchmark reports
If you publish numbers, include at minimum:
- Stage-by-stage p50/p95/p99, not just end-to-end.
- Hardware and network profile (localhost, same-region cloud, cross-continent).
- ASR model and whether streaming partials were used.
- LLM model, context size, and streaming config.
- TTS model and time-to-first-byte measurement.
- Fallback simulation results with forced provider errors.
- Audio sample characteristics (silence padding, noise floor, length).
Without that, the benchmark is marketing, not engineering.
Decisive takeaway
Treat speech-to-speech latency benchmark design as a profiling exercise, not a leaderboard. Break the pipeline into ASR, LLM TTFT, LLM completion, and TTS first-audio; measure each under fault injection; and overlap stages with streaming. Stacks that look identical in a naive end-to-end test diverge violently once you isolate the stages and simulate a provider hiccup. Build the harness, collect the percentiles, and optimize the worst stage—not the headline number.