Speech-to-speech apps with the OpenAI Realtime API

Building a voice agent that talks back in real time used to mean stitching together ASR, LLM, and TTS services with brittle glue code. This openai realtime api speech-to-speech tutorial walks through a working pattern for a single WebSocket session that streams audio both ways, using OpenAI’s Realtime API directly from the browser or a server. We’ll cover session setup, audio handling, and the failure modes that bite in production.

1. Open the Realtime WebSocket correctly

The Realtime API is a single WebSocket at wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01. Authentication uses a bearer token in the Authorization header. Client libraries are thin; you mostly send and receive JSON events.

const url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01";
const ws = new WebSocket(url, {
  headers: { Authorization: `Bearer ${API_KEY}` },
});
ws.on("open", () => console.log("realtime socket open"));

Never embed API_KEY in shipped browser code. We’ll fix that in section 6. For local prototyping, an env-injected key is fine.

2. Configure the session before streaming audio

Before sending audio, send a session.update event. The model will not accept audio until it knows the format. Use PCM16 at 24 kHz mono for both directions—this matches the API’s expectations and avoids resampling artifacts.

{
  "type": "session.update",
  "session": {
    "modalities": ["audio", "text"],
    "instructions": "You are a concise voice assistant. Speak naturally.",
    "voice": "alloy",
    "input_audio_format": "pcm16",
    "output_audio_format": "pcm16",
    "input_audio_transcription": { "model": "whisper-1" },
    "turn_detection": { "type": "server_vad", "silence_duration_ms": 500 }
  }
}

Set turn_detection to server_vad so the API detects speech boundaries. The silence_duration_ms trades responsiveness against false interruptions.

3. Capture microphone audio in the browser

getUserMedia returns a MediaStream. You need raw samples, not Opus blobs. Create an AudioContext at 24 kHz and tap the stream with a ScriptProcessorNode (or an AudioWorklet if you want to avoid deprecation warnings). Convert Float32 to Int16, base64-encode, and append to the input buffer.

const ctx = new AudioContext({ sampleRate: 24000 });
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const src = ctx.createMediaStreamSource(stream);
const processor = ctx.createScriptProcessor(4096, 1, 1);

processor.onaudioprocess = (e) => {
  const float = e.inputBuffer.getChannelData(0);
  const int16 = new Int16Array(float.length);
  for (let i = 0; i < float.length; i++) {
    int16[i] = Math.max(-1, Math.min(1, float[i])) * 0x7fff;
  }
  const b64 = btoa(String.fromCharCode(...new Uint8Array(int16.buffer)));
  ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: b64,
  }));
};
src.connect(processor);
processor.connect(ctx.destination);

Call ctx.resume() inside a click handler—browsers block audio until a user gesture.

4. Play model audio without clicks

The API streams response.audio.delta events with base64 PCM16 chunks. Do not create a new AudioBufferSourceNode per chunk; that causes audible gaps. Instead, push decoded samples into a queue and drain them from a single ScriptProcessorNode connected to ctx.destination.

const playQueue: Int16Array[] = [];

ws.onmessage = (ev) => {
  const msg = JSON.parse(ev.data);
  if (msg.type === "response.audio.delta") {
    const bin = atob(msg.audio);
    const samples = new Int16Array(bin.length / 2);
    for (let i = 0; i < samples.length; i++) {
      samples[i] = (bin.charCodeAt(i * 2) | (bin.charCodeAt(i * 2 + 1) << 8));
    }
    playQueue.push(samples);
  }
};

const sink = ctx.createScriptProcessor(4096, 1, 1);
sink.onaudioprocess = (e) => {
  const out = e.outputBuffer.getChannelData(0);
  let written = 0;
  while (playQueue.length && written < out.length) {
    const chunk = playQueue[0];
    for (let i = 0; i < chunk.length && written < out.length; i++) {
      out[written++] = chunk[i] / 0x7fff;
    }
    if (written >= out.length) break;
    playQueue.shift();
  }
};
sink.connect(ctx.destination);

This ring-buffer approach adds ~100 ms latency but stays glitch-free on Chrome and Safari.

5. Manage turn-taking and interruptions

Server VAD emits input_audio_buffer.speech_stopped when the user finishes. Trigger a response with response.create:

ws.send(JSON.stringify({ type: "response.create" }));

If the user starts speaking again mid-response, the API sends input_audio_buffer.speech_started. Send response.cancel to stop the model and free the audio output. This barge-in behavior is expected in voice UX but can feel abrupt. Tune silence_duration_ms in section 2 to reduce accidental cutoffs.

Transcription arrives as conversation.item.input_audio_transcription.completed. Use it for logging or grounding, not for low-latency decisions—it lags the audio.

6. Proxy the socket to hide your key

Ship a minimal relay. The browser connects to your server; the server opens the OpenAI socket with the secret. Python with websockets:

import os
import asyncio
import websockets

OPENAI_KEY = os.environ["OPENAI_KEY"]

async def relay(browser_ws):
    async with websockets.connect(
        "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01",
        extra_headers={"Authorization": f"Bearer {OPENAI_KEY}"},
    ) as api_ws:
        async def forward_browser_to_api():
            async for msg in browser_ws:
                await api_ws.send(msg)
        async def forward_api_to_browser():
            async for msg in api_ws:
                await browser_ws.send(msg)
        await asyncio.gather(forward_browser_to_api(), forward_api_to_browser())

Run this behind TLS. The relay adds one network hop; place it near your users to keep latency under 200 ms round-trip.

7. Common pitfalls and tradeoffs

Autoplay policy. Safari and Chrome require a user gesture before AudioContext starts. Render a “Start call” button that calls ctx.resume().

Sample rate mismatch. If you capture at 48 kHz and lie about it in session.update, you get chipmunk audio. Resample explicitly or set ctx.sampleRate = 24000.

Cost visibility. Realtime billing is per audio minute, not tokens. Instrument your relay to count connected seconds; otherwise finance will be surprised.

Model coverage. Only gpt-4o-realtime-preview variants support this endpoint. You cannot use standard gpt-4o here.

Transcription limits. whisper-1 handles common accents but fails on heavy noise. Don’t rely on it for safety-critical input.

Connection drops. WebSocket disconnects are silent. Implement exponential backoff and re-create the session, including re-sending session.update.

8. Hybrid architectures

A pure voice loop is rarely enough. You’ll want to run background summarization, RAG, or tool calls that don’t need realtime audio. For those, route standard chat completions to an OpenAI-compatible gateway such as n4n.ai, which addresses 240+ models with automatic fallback when a provider degrades and per-token metering. Keep the Realtime WebSocket exclusive to the voice path to avoid head-of-line blocking.

9. Test with synthetic audio

Don’t debug with your own microphone. Generate a 24 kHz PCM16 sine sweep, base64 it, and feed input_audio_buffer.append in a loop. Assert that you receive response.audio.delta and that playback queue drains. This catches format bugs faster than shouting at your laptop.

The openai realtime api speech-to-speech tutorial above is a baseline. Ship the relay, measure latency on real devices, and treat turn-detection tuning as a product decision, not a config afterthought.