Building a voice agent forces a fork in the road: pipe audio through separate transcription, language, and synthesis models, or hand it to a single end-to-end model. The gap in cascaded vs native speech-to-speech latency is the deciding factor for most real-time products, but speed is not the only axis that breaks differently. You also trade control, cost granularity, and debuggability.
Architecture basics
Cascaded pipeline
A cascaded stack chains three independent services: automatic speech recognition (ASR), a text LLM, and text-to-speech (TTS). Each stage runs as its own network call or local model.
# Typical cascaded flow with OpenAI-compatible endpoints
import openai
audio_file = open("user_clip.webm", "rb")
transcript = openai.audio.transcriptions.create(
model="whisper-1", file=audio_file
).text
llm_resp = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": transcript}]
)
reply_text = llm_resp.choices[0].message.content
speech = openai.audio.speech.create(
model="tts-1",
input=reply_text,
voice="alloy"
)
# speech.content holds MP3 bytesNative speech-to-speech
A native model accepts raw audio frames and emits audio frames over a persistent connection. There is no exposed text transcript unless the API explicitly echoes it.
# Native via OpenAI Realtime API (truncated)
import websockets, json, base64
async def talk():
async with websockets.connect(
"wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
) as ws:
await ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(pcm_chunk).decode()
}))
# Server streams response.audio.delta frames backCapabilities
Cascaded wins on composability. You can run Whisper for ASR, a reasoning-optimized LLM for the brain, and a cloned voice in ElevenLabs for output. Swapping any layer does not require retraining. You get the text transcript for free, which makes logging, moderation, and RAG trivial.
Native models trade that transparency for tighter coupling between understanding and expression. They handle backchanneling, prosody, and mid-sentence interruptions with less code because the model itself decides when to start speaking. What you lose is the ability to inspect or edit the “thought” text before it becomes voice.
Cost model
Cascaded billing is itemized. You pay per audio minute for ASR, per input/output token for the LLM, and per character or per audio second for TTS. At scale this lets you optimize each line item—use a cheap ASR for clean studio audio, a premium TTS only for the final response.
Native pricing is usually bundled: a per-minute or per-audio-token rate that covers the whole pipeline. It can be simpler to predict, but you cannot selectively downgrade one stage. If the model spends 10 seconds “thinking” in audio before replying, you pay for that silence.
Latency and throughput
The core of this post is cascaded vs native speech-to-speech latency, and the numbers follow from the architecture. A cascaded system serializes three cold starts. Native overlaps perception and generation.
Breaking down the latency budget
In a cascaded stack, time-to-first-audio (TTFA) is roughly:
- ASR time-to-first-transcript: 150–500 ms depending on endpoint and streaming config
- LLM time-to-first-token: 100–400 ms for small models, more for large
- TTS time-to-first-byte: 80–250 ms for neural vocoders
Summed, you routinely see 400–1200 ms before the user hears anything. Native models collapse this: a single forward pass produces audio tokens in 200–500 ms typical, because the model does not wait for a finalized text transcript. The cascaded vs native speech-to-speech latency difference is therefore often a 2–3x gap in TTFA.
Streaming and interruption handling
Cascaded stacks can fake low latency by streaming LLM tokens into an incremental TTS (sentence-boundary or token-bucket synthesis). That helps, but the ASR stage still must commit to a final hypothesis before the LLM starts. Barge-in (user interrupts) requires canceling TTS and flushing the LLM—doable, but you write that logic yourself.
Native endpoints treat interruption as a first-class signal: send a new audio buffer, the model stops its output stream. That behavior is built in, which is why the cascaded vs native speech-to-speech latency gap feels even larger in chaotic conversational settings.
Ergonomics and developer experience
Cascaded is more files. You manage three clients, three retry policies, and three rate limits. The upside: when audio quality drops, you know exactly which layer failed. A 500 from TTS does not kill your transcript.
Native is a single WebSocket and a binary frame protocol. Less glue code, but when the model returns garbled audio you have no intermediate text to diff. Debugging means replaying audio into a black box.
When you run a cascaded stack across multiple vendors, an OpenAI-compatible gateway such as n4n.ai can consolidate billing and automatically fall back if a provider is degraded, while forwarding cache-control hints to cut repeat ASR costs on short repeated phrases.
Ecosystem and model availability
Cascaded has depth: dozens of ASR engines (Whisper, Deepgram, Azure), thousands of LLMs, and many TTS voices. You can mix open-weight and proprietary freely.
Native speech-to-speech is still early. Production-grade options are limited to a handful of closed APIs (OpenAI Realtime, Gemini Live, Sesame). Open-weight native models exist in research but lack the serving infrastructure most teams need. If you need Urdu ASR with a custom Welsh voice, cascaded is your only realistic path today.
Operational limits
Cascaded failure modes are independent but additive: ASR hallucination poisons the LLM, LLM drift produces text the TTS cannot pronounce. Latency compounds under load because each stage queues separately.
Native models fail holistically. A degraded model slows both comprehension and speech. You cannot route around a bad TTS because it is the same tensor. Language coverage is narrower, and you are locked to the vendor’s audio format and voice roster.
Head-to-head comparison
| Dimension | Cascaded (ASR+LLM+TTS) | Native speech-to-speech |
|---|---|---|
| Capabilities | Modular, full text access, easy RAG/moderation | Unified prosody, built-in barge-in, no transcript |
| Cost model | Itemized per stage; optimize each | Bundled per minute/token; pay for silence |
| Latency/throughput | Serialized 400–1200 ms TTFA; scalable per stage | Overlapped 200–500 ms TTFA; single bottleneck |
| Ergonomics | More code, isolated failures, debuggable | Single socket, less code, opaque errors |
| Ecosystem | Broad ASR/LLM/TTS choice, open weights | Few closed APIs, limited languages/voices |
| Limits | Error propagation, latency stacking | Vendor lock, no intermediate text, narrow coverage |
Which to choose
Use native when…
You are building a consumer voice chat where perceived latency beats accuracy. A conversational coach, a casual companion, or a drive-thru order taker benefits from sub-500 ms responses and natural interruptions. You can tolerate less control over exact wording and voice.
Use cascaded when…
You need audit trails, custom voices, or support for languages the native models ignore. Call-center analytics, medical dictation, and any workflow that feeds the transcript into a database should stay cascaded. The extra 300 ms is irrelevant next to compliance.
Hybrid approach
Run native for the live conversational loop, but silently fork the audio to an ASR+LLM cascade for logging and post-call summarization. This captures the latency win without surrendering the text. The cascaded vs native speech-to-speech latency tradeoff is not binary—most production systems will land here within a year.