GPT-4o Realtime API latency benchmark for voice agents

A credible gpt-4o realtime api latency benchmark has to separate the time the model spends thinking from the time your users spend waiting for a voice to answer. Most published numbers mash connection setup, speech detection, and audio playback into a single “latency” figure, which hides the levers you can actually pull. This analysis breaks the pipeline into measurable stages and shows where the Realtime API wins—and where it won’t save you from bad client architecture.

What the Realtime API actually does

The GPT-4o Realtime API is a single WebSocket session that accepts raw audio in, runs speech-to-text, language model inference, and text-to-speech out, all streamed. You open one connection, send a session.update to configure voice and turn detection, then stream microphone bytes. The server sends response.audio.delta frames you play directly. There is no separate ASR call, no HTTP POST per turn.

That design collapses three network round trips (ASR → LLM → TTS) into one persistent connection. The theoretical win is large: instead of waiting for a full utterance transcript before the LLM starts, the model receives tokens as they are recognized and begins generating audio while you are still talking. In practice, server-side turn detection still gates generation on a silence threshold, so the clock starts when the user stops speaking.

Breaking down latency components

To talk about a gpt-4o realtime api latency benchmark honestly, label each stage:

Capture and VAD (client-side): Microphone capture, jitter buffer, and local voice activity detection if you override server VAD.
Transport to OpenAI: TLS + WebSocket overhead, plus physical distance to the region.
Server ingest and ASR: Converting incoming audio to text tokens.
Model prefill and decode: GPT-4o processing context and emitting response tokens.
TTS synthesis: Converting response tokens to audio frames.
Transport back: WebSocket delivery to client.
Playback buffer: Client audio sink latency and buffering for smooth output.

Stages 3–5 happen server-side and are outside your control beyond model choice. Stages 1, 2, 6, 7 are yours. The gpt-4o realtime api latency benchmark that matters is the interval from speech end (detected silence) to first audible audio frame at the speaker—call it time-to-first-audio (TTFA).

How to measure it without lying to yourself

Most teams measure from when they think the user stopped talking to when they receive the first response.audio.delta. That ignores playback buffering and network jitter. Worse, they include the initial WebSocket handshake in the median, which is a one-time cost.

WebSocket setup and session warmup

Open the connection once and keep it. A cold connect in us-east-1 to OpenAI adds 100–300ms of TLS and protocol negotiation; that is irrelevant to conversational turns but dominates a naive “first message” benchmark. Warm up by sending a trivial session update and waiting for session.updated before starting the clock.

import asyncio, json, time
import websockets

async def warmup(uri, api_key):
    async with websockets.connect(
        uri,
        additional_headers={"Authorization": f"Bearer {api_key}"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"turn_detection": {"type": "server_vad"}}
        }))
        async for msg in ws:
            if json.loads(msg).get("type") == "session.updated":
                return time.monotonic()

Instrumenting the client

Stamp time.monotonic() at the moment your client detects end-of-speech (or when the server sends input_audio_buffer.speech_stopped). Then stamp when you hand the first audio delta to the sound card. Difference is TTFA. Do not use wall-clock datetime—you need monotonic resolution.

# pseudo-client snippet
speech_end = None
first_audio_played = None

async def on_message(msg):
    global speech_end, first_audio_played
    data = json.loads(msg)
    if data["type"] == "input_audio_buffer.speech_stopped" and speech_end is None:
        speech_end = time.monotonic()
    if data["type"] == "response.audio.delta" and first_audio_played is None:
        first_audio_played = time.monotonic()
        audio_sink.play(data["delta"])
        print(f"TTFA: {(first_audio_played - speech_end)*1000:.1f} ms")

Run at least 200 turns across varied utterance lengths. Report median and p95, not average.

Benchmark methodology (illustrative)

A defensible gpt-4o realtime api latency benchmark uses a synthetic speaker (pre-recorded clips) played into a virtual audio device, eliminating human variability. Script the session:

Region: same as your production deployment (e.g., wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01).
Network: modest 40ms RTT, 10mbps shaped link to mimic residential broadband.
Utterance set: 20 short commands (“turn off the lights”), 20 medium (“what’s the weather in Boston tomorrow”), 20 long (“explain quantum computing like I’m ten”).
Measure TTFA and also full-turn latency (speech end to last audio frame delivered).

You will find TTFA is dominated by server-side TTS onset, which is tightly coupled to model decode speed. The LLM prefill for a 2k-token context is sub-100ms; the audio synthesis adds the visible delay. That is why switching to a smaller model rarely changes TTFA dramatically—the realtime pipeline is already optimized for streaming audio.

Tradeoffs vs. a modular pipeline

The alternative is a discretized stack: Whisper for ASR, a chat completion for LLM, a TTS engine like Azure or Cartesia. You control each stage and can swap a faster TTS. But you pay three round trips and must build barge-in handling yourself.

Against that, the Realtime API gives you:

Single connection, built-in turn taking, and barge-in via response.cancel.
No orchestration code for ASR→LLM→TTS.
Consistent voice model tuned for low latency.

The cost: you are locked to OpenAI’s voice synthesis latency and cannot use a custom TTS with lower MOS-but-faster output. If your product needs a specific celebrity voice or on-device TTS, Realtime API is not suitable.

For a gpt-4o realtime api latency benchmark to inform architecture, run both: measure TTFA for Realtime, then measure ASR+LLM+TTS with your chosen components. In our tests, modular stacks with local VAD and a fast TTS can match TTFA but complicate state management. Realtime wins on engineering effort.

Where a gateway fits

If your voice agent also calls non-realtime endpoints—say, a post-call summary or a tool-calling step that doesn’t need audio—fronting those with a gateway such as n4n.ai gives you automatic fallback across providers without client changes. The Realtime session itself stays direct to OpenAI; mixing the two is clean because the audio path is isolated.

That said, do not route the Realtime WebSocket through a non-native proxy unless it supports streaming binary frames with low overhead. Adding a hop there will inflate TTFA by your proxy’s region distance, defeating the purpose.

Takeaway

The gpt-4o realtime api latency benchmark that should drive your decision is time-to-first-audio from speech stop, measured on warm connections with monotonic clocks. The model is not your bottleneck; your client’s audio buffer and the unavoidable TTS synthesis are. Default to the Realtime API for interactive voice unless you need a custom TTS or on-premise data residency. Benchmark it correctly, keep the socket warm, and spend your optimization cycles on client-side playback jitter, not model shopping.