Measuring interruption latency in real-time voice agents

Interruption latency voice agents experience is the time from a user starting to talk over the agent to the moment the agent’s speech actually stops. It is the single most perceptible quality signal in a duplex voice conversation, yet most teams track it indirectly through ASR or LLM benchmarks. The thesis here is simple: you cannot optimize what you have not measured end-to-end, and the component numbers you are quoting are lying to you.

What interruption latency actually measures

The clock starts at acoustic onset—the first milliseconds of user speech that overlap the agent’s outgoing audio. It stops when the speaker in the user’s device goes silent from the agent’s stream. Everything in between is fair game:

• Microphone capture and client-side buffering
• Voice activity detection (VAD) on the incoming stream
• Optional endpointing or ASR finalization
• Turn-taking logic (should we really interrupt?)
• LLM or dialog policy inference for the new turn
• TTS synthesis start and the cancellation of in-flight audio
• Network round-trip and client audio playback drain

If you measure only the ASR delay or only the TTS cancel RPC, you are ignoring the dominant contributors: client buffer depth and VAD conservatism.

Why component benchmarks lie

A VAD library might claim 15 ms framing latency. An ASR service might advertise 200 ms time-to-first-token. A TTS provider might stream the first chunk in 80 ms. Add them and you expect a small number. In production we routinely see interruption latency voice agents ship with that blows past the sum—often crossing a full second before the agent yields, because those numbers omit:

1. Audio interface buffer sizes (often 20–100 ms per device)
2. WebSocket or WebRTC jitter buffers
3. The time to flush already-queued TTS frames from the client sound card
4. The policy delay where the system waits for ASR stability before committing to interrupt

Worse, teams celebrate a faster LLM while the agent keeps talking for half a second because the cancel signal was never sent until after the LLM responded.

Instrumenting the pipeline

You need timestamps at every stage, emitted as events the client and server both log. Below is a minimal asyncio WebSocket monitor that assumes your server tags user_speech_start and agent_audio_stop. It is not a fake API; it is the shape you should adopt.

import asyncio, json, time

async def monitor(ws):
    start = None
    async for raw in ws:
        ev = json.loads(raw)
        if ev["type"] == "user_speech_start" and start is None:
            start = time.monotonic()
        elif ev["type"] == "agent_audio_stop" and start is not None:
            latency_ms = (time.monotonic() - start) * 1000
            print(f"interruption latency: {latency_ms:.1f} ms")
            start = None

Run this against a simulated barge-in load. The printout is your ground truth. Do not trust a number derived from server logs alone—the audio stop event must be captured at the client’s audio sink, not at the server’s send socket.

VAD tuning and its tradeoffs

The fastest lever is VAD aggressiveness. Using webrtcvad as an example:

import webrtcvad

vad = webrtcvad.Vad(3)  # mode 3 = most aggressive
frame = pcm_20ms_16k()   # bytes, 16-bit mono
if vad.is_speech(frame, 16000):
    trigger_interrupt()

Mode 3 cuts detection to a single 20 ms frame but will false-trigger on background noise, coughing, or the agent’s own echo if acoustic echo cancellation is weak. Mode 0 is safe but can add hundreds of milliseconds as it waits for sustained speech. For interruption latency voice agents, I recommend mode 2 or 3 with a short confirmation window of 40 ms and solid AEC. The cost is occasional premature cuts; the benefit is the user feels heard.

Echo cancellation is non-negotiable

If the agent’s speech leaks into the mic, an aggressive VAD will interrupt itself. Measure your AEC tail length. A software AEC like SpeexDSP typically needs 100–150 ms tail; WebRTC’s AEC3 handles it better but costs CPU. Skip this and no VAD tuning saves you.

Cancellation and TTS buffering

Once interrupted, you must stop the synth. With streaming TTS, send a cancel and drop queued frames:

{"type": "tts_cancel", "request_id": "req-123"}

If your TTS vendor does not support cancellation, you are at the mercy of the client buffer. Even with cancel, the client audio stack may have 200 ms of buffered PCM. Drain it explicitly:

stream.stop_stream()
stream.write(b'\x00' * 1024)  # optional silence pad to flush

The point: interruption latency voice agents report is the sum of detection, decision, cancel propagation, and playback drain. You can only shave the last part by tracking the client’s exact buffered sample count.

Network and client rendering

Measure at the speaker, not the socket. A typical browser Web Audio buffer is 256–1024 samples at 48 kHz (~5–21 ms), but the OS mixer adds another 10–30 ms. On native mobile, AudioUnit latency can be sub-10 ms if you bypass the high-level API. If you cannot instrument the sink, place a physical microphone and a second device to record both channels and align waveforms. That is low-tech but definitive.

Where the LLM fits

After barge-in, the agent needs a new utterance. This is where inference speed matters, but it is downstream of the interrupt. Once the barge-in is detected, the agent needs a new response; an inference gateway such as n4n.ai that honors client routing directives and forwards provider cache-control hints can reduce time-to-first-token, but that sits downstream of the interruption latency voice agents care about. Optimizing LLM latency from 800 ms to 400 ms is worthless if the user already waited 900 ms for the agent to shut up.

A reference measurement setup

Build a loopback harness:

1. Agent audio plays from speaker A.
2. Microphone B feeds the agent’s VAD.
3. A script injects a loud tone at T=0 to simulate user speech.
4. A recorder captures speaker A’s output and the tone.
5. Cross-correlate to find when agent amplitude drops below threshold.

ffmpeg -f pulse -i default -t 5 out.wav &
sleep 1 && play tone.wav   # inject barge-in

Analyze with sox or python scipy to find silence onset. This gives you an honest number without trusting internal logs.

Tradeoffs weighed

• Early VAD: low latency, more false cuts.
• Late VAD: stable turns, sluggish feel.
• Client-side cancel: precise, needs native code.
• Server-side only: simpler, but playback buffer hides true stop.
• No TTS cancel API: forced to wait for buffer drain.

Pick early VAD + client cancel every time for consumer voice UX. The occasional awkward cut is less damaging than a bot that talks over you.

Decisive takeaway

Measure interruption latency voice agents end-to-end from acoustic onset to speaker silence, instrument every stage with monotonic timestamps, and tune VAD aggressive enough to trigger within 40 ms while relying on AEC to prevent self-interruption. Treat LLM speed as a separate concern. If you do only one thing this week, log user_speech_start and agent_audio_stop on the client and watch the distribution—not the average, the p95. That number is your product.