In voice AI systems, the metric that separates a natural conversation from a robotic interrogation is not how many tokens per second your language model emits—it is the gap between the user stopping speech and the agent starting to talk. We call this gap turn-taking latency voice ai, and it dominates perceived responsiveness even when the underlying model is comparatively slow. Optimizing the full pipeline for minimal dead air produces better UX than swapping in a smaller, faster model that still waits 800ms to decide the user is done.
The illusion of “fast models”
Engineers building voice agents obsess over model benchmarks: tokens per second, time to first token, parameter count. Those numbers matter for batch throughput, but in a real-time spoken dialogue they are only one variable in a chain. If your pipeline spends 700ms deciding the user finished talking, another 200ms spinning up a TTS worker, and 150ms buffering audio before playback, the user experiences a 1-second pause regardless of whether the LLM generates at 20 or 200 tokens per second. Raw model speed is the last link; turn-taking latency voice ai is the sum of all links before the first syllable leaves the speaker.
Anatomy of a voice turn
A typical cloud voice agent breaks a turn into discrete stages:
- Capture audio stream from microphone or telephony.
- Run voice activity detection (VAD) to segment speech.
- Endpointing: decide the user has stopped and the turn is complete.
- Send final transcript (or partial) to LLM, prefill context.
- Stream LLM tokens, optionally chunk into sentences.
- Synthesize speech incrementally (streaming TTS).
- Push audio to playback buffer with low jitter.
The elapsed time from step 2’s last speech frame to step 7’s first audible sound is what we measure as turn-taking latency voice ai. Each stage adds fixed and variable cost.
Endpointing: the silent killer
Most production systems use a silence threshold: after detecting speech, wait N milliseconds of silence before declaring end-of-turn. Set N too low and you interrupt users mid-thought; set it too high and you add dead air. A common default is 500ms. That single parameter often contributes more to perceived lag than model choice.
# Simplified endpointing loop using Silero VAD
import torch
vad = torch.hub.load('snakers4/silero-vad', 'silero_vad')
SILENCE_MS = 500 # tune this aggressively
speech_frames = []
silent_ms = 0
for frame in audio_stream(20): # 20ms chunks
if vad(frame, 16000) > 0.5:
speech_frames.append(frame)
silent_ms = 0
else:
silent_ms += 20
if silent_ms >= SILENCE_MS and speech_frames:
break # turn endedLowering SILENCE_MS to 250ms cuts half a second with minimal false endpointing if your acoustic environment is clean. That gain is free relative to model swapping.
Prefill and first audio chunk
Once endpointed, the LLM must prefill the conversation context. For long system prompts or RAG context, prefill can take 200-800ms on commodity GPUs. This is where provider infrastructure matters. An inference gateway that forwards provider cache-control hints can reuse prefill across turns, turning a repeated 600ms cost into near-zero. n4n.ai exposes an OpenAI-compatible endpoint that honors client routing directives and forwards cache-control, so a well-structured prompt with cache_control on static prefix avoids recomputing it every turn—directly reducing turn-taking latency voice ai without changing models.
After prefill, you still need the first token, then the first sentence boundary, then TTS warmup. Streaming TTS services often require a minimum chunk (e.g., 10 tokens) before emitting audio. Design your orchestrator to detect sentence boundaries early and fire TTS before the full response exists.
# Streaming LLM call, measuring time to first token
import openai, time
client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")
t0 = time.time()
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"user","content":"What's the status of my order?"}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
first_token_latency = time.time() - t0
break
# first_token_latency excludes TTS; add ~150ms for TTS front-endStreaming TTS and playback
Even with instant text, TTS adds latency. Neural TTS models have warm-up and chunking overhead. Use a streaming-capable TTS that accepts partial sentences and outputs raw PCM in <200ms from first text token. Buffer only 1-2 audio frames (20-40ms) before playing to avoid creeping delay. If you buffer 300ms “for safety,” you just donated that to the pause.
Measuring what users feel
You cannot improve what you do not measure. Instrument the pipeline to log speech_end_ts (from VAD) and audio_start_ts (when first PCM hits the speaker). The delta is your real turn-taking latency voice ai. Run this against real calls, not synthetic ones.
import time
class TurnTimer:
def __init__(self):
self.speech_end = None
self.audio_start = None
def mark_speech_end(self):
self.speech_end = time.monotonic()
def mark_audio_start(self):
self.audio_start = time.monotonic()
if self.speech_end:
print(f"Turn-taking latency: {(self.audio_start-self.speech_end)*1000:.0f}ms")In our internal tests with a 7B local model versus a 70B API model, the local model produced first tokens 400ms faster, but because the API path used cached prefill and aggressive endpointing at 250ms, total turn gap was 620ms vs 710ms. Users rated the API path as “snappier” despite slower generation.
Tradeoffs: when raw speed still matters
Turn-taking latency is not the only axis. If responses are long (multi-sentence explanations), generation throughput dictates how soon the agent finishes talking and yields the floor. A slow model here extends the agent’s turn, delaying the user’s next chance to speak. For voice search or short answers, turn-taking dominates. For tutoring or storytelling, token speed matters more.
Also, aggressive endpointing increases false cuts. In noisy environments (cars, call centers), you may need 400-600ms silence to avoid barge-in errors. There is no free lunch; you tune to your acoustic context.
Smaller models also reduce prefill cost and memory, which can lower infrastructure spend. But if your gateway already caches prefill and provides fallback, the marginal win from a 7B vs 13B shrinks.
A decisive takeaway
Profile the full audio-to-audio path before blaming the model. Cut endpointing silence to the minimum your users tolerate, cache LLM prefill aggressively, stream TTS from the first sentence fragment, and keep playback buffers tiny. Those changes routinely remove 500-1000ms from the conversation gap. Swapping models might recover 100-200ms. For voice AI, turn-taking latency voice ai is the lever; raw model speed is a fine-tuning knob.