How real-time speech-to-text streaming works

Real-time speech-to-text streaming moves audio from a microphone to a transcription engine and back to the client in under 300 milliseconds end-to-end. The hard part isn’t the model — it’s the plumbing: chunking, buffering, backpressure, and keeping the connection alive when networks wobble. This guide walks through the architecture, the protocol choices, and the failure modes you’ll hit in production.

Choose your transport: websockets vs http/2 vs grpc

WebSockets remain the default for browser clients. They’re bidirectional, widely supported, and work through most corporate proxies. The tradeoff: you own heartbeat logic, reconnection, and message framing.

HTTP/2 (or HTTP/3) with server-sent events (SSE) works when you only need server-to-client streaming. Simpler retry semantics, but no client-to-server audio upload on the same connection — you’ll need a second POST stream or chunked upload.

gRPC streaming RPCs give you typed contracts, flow control, and cancellation baked in. Best for service-to-service inside your infrastructure. Browser support requires gRPC-Web and a proxy (Envoy), adding operational complexity.

// speech.proto
syntax = "proto3";

package speech.v1;

service SpeechToText {
  rpc StreamingRecognize(stream StreamingRecognizeRequest)
       returns (stream StreamingRecognizeResponse);
}

message StreamingRecognizeRequest {
  oneof audio_source {
    RecognitionConfig config = 1;
    bytes audio_content = 2;
  }
}

message RecognitionConfig {
  string model = 1;
  string language_code = 2;
  int32 sample_rate_hertz = 3;
  bool enable_automatic_punctuation = 4;
  bool interim_results = 5;
}

message StreamingRecognizeResponse {
  repeated SpeechRecognitionResult results = 1;
  bool is_final = 2;
  int32 stability = 3; // 0-100, provider-specific
}

message SpeechRecognitionResult {
  string transcript = 1;
  float confidence = 2;
  bool is_final = 3;
  repeated WordInfo words = 4;
}

message WordInfo {
  string word = 1;
  float start_time = 2; // seconds
  float end_time = 3;
  float confidence = 4;
}

Pitfall: WebSocket frames have a 125-byte overhead per frame for small payloads. Batch 100–200 ms of audio per frame to amortize it. Don’t send 20 ms frames — you’ll saturate the event loop.

Chunking strategy: fixed vs vad vs hybrid

Fixed-size chunks (100 ms, 200 ms) are trivial to implement. The model receives uniform input, but you waste compute on silence and add latency waiting for the buffer to fill.

Voice activity detection (VAD) sends audio only when speech is present. Reduces compute and bandwidth, but introduces jitter: the VAD decision itself takes 20–50 ms, and you need a pre-speech padding buffer (typically 300 ms) so the model gets context before the first voiced frame.

Hybrid approach: fixed chunks with VAD gating. Accumulate 100 ms frames, run a lightweight VAD (Silero, WebRTC VAD) on each, and only forward frames above threshold. Keep a ring buffer of the last 3–5 frames for pre-speech context.

# vad_gate.py
import numpy as np
import webrtcvad

SAMPLE_RATE = 16000
FRAME_MS = 100
FRAME_BYTES = int(SAMPLE_RATE * FRAME_MS / 1000) * 2  # 16-bit mono

vad = webrtcvad.Vad(2)  # aggressiveness 0-3
pre_speech_buffer = []
MAX_PRE_SPEECH = 3  # frames

def process_chunk(pcm_bytes: bytes) -> list[bytes]:
    """Return frames to forward, or empty list if gated."""
    global pre_speech_buffer
    frames = []
    
    for i in range(0, len(pcm_bytes), FRAME_BYTES):
        frame = pcm_bytes[i:i+FRAME_BYTES]
        if len(frame) < FRAME_BYTES:
            break  # partial frame, wait for more
        
        is_speech = vad.is_speech(frame, SAMPLE_RATE)
        
        if is_speech:
            # flush pre-speech buffer
            frames.extend(pre_speech_buffer)
            pre_speech_buffer.clear()
            frames.append(frame)
        else:
            pre_speech_buffer.append(frame)
            if len(pre_speech_buffer) > MAX_PRE_SPEECH:
                pre_speech_buffer.pop(0)
    
    return frames

Tradeoff: Aggressive VAD (level 3) drops quiet speakers. Level 1 lets through background noise. Test with your actual acoustic environments — meeting rooms, open offices, cars — before picking a threshold.

Backpressure and flow control

The audio capture rate (16 kHz × 16-bit mono = 32 KB/s) is constant. The recognition pipeline varies: model inference, network RTT, provider queue depth. Without backpressure, memory grows until OOM.

WebSocket: monitor socket.bufferedAmount. Pause microphone capture when it exceeds a high-water mark (e.g., 500 KB), resume at low-water (100 KB).

gRPC: use built-in flow control. The client respects WINDOW_UPDATE frames from the server. Set initial window to 1 MB, don’t rely on defaults.

# backpressure_ws.py
import asyncio
import websockets

HIGH_WATER = 500_000  # bytes
LOW_WATER = 100_000

class BackpressuredSender:
    def __init__(self, ws):
        self.ws = ws
        self.paused = False
        self._monitor_task = None
    
    async def start(self):
        self._monitor_task = asyncio.create_task(self._monitor())
    
    async def _monitor(self):
        while True:
            await asyncio.sleep(0.05)  # 50 ms poll
            buffered = self.ws.transport.get_write_buffer_size()
            if not self.paused and buffered > HIGH_WATER:
                self.paused = True
                self.on_pause()
            elif self.paused and buffered < LOW_WATER:
                self.paused = False
                self.on_resume()
    
    def on_pause(self):
        # signal audio capture to stop
        pass
    
    def on_resume(self):
        # signal audio capture to resume
        pass
    
    async def send(self, data: bytes):
        await self.ws.send(data)
    
    async def close(self):
        if self._monitor_task:
            self._monitor_task.cancel()

Pitfall: Browser MediaRecorder doesn’t expose backpressure. You must implement it at the AudioWorklet level — push audio into a ring buffer, have the Worklet pull only when the sender signals ready.

Interim results and finalization

Streaming recognition returns interim (unstable) hypotheses that update as more context arrives. UX expects these to appear character-by-character. The protocol must distinguish interim from final.

Typical pattern: each response carries is_final per result. Interim results have is_final=false and may have a stability score (0–100). Final results have is_final=true and won’t change.

// client-side result handling
let interimTranscript = '';
let finalTranscript = '';

ws.onmessage = (event) => {
  const response = JSON.parse(event.data);
  
  for (const result of response.results) {
    if (result.is_final) {
      finalTranscript += result.transcript + ' ';
      interimTranscript = '';
      renderFinal(finalTranscript);
    } else {
      interimTranscript = result.transcript;
      renderInterim(finalTranscript + interimTranscript, result.stability);
    }
  }
};

Pitfall: Some providers send a new interim result for every chunk, even when the hypothesis hasn’t changed. Dedup on the client: only re-render when transcript or stability actually differs.

Endpointing: when to finalize

Endpointing decides when an utterance ends. Too eager → mid-sentence cuts. Too lazy → user waits for final result.

Server-side endpointing: the provider watches for trailing silence (typically 0.8–1.5 s configurable). Returns is_final=true automatically. Simplest for clients.

Client-side endpointing: you control the silence threshold. Send an explicit END_OF_STREAM message or close the stream. Required for push-to-talk or when you need tighter latency.

Hybrid: use server endpointing for dictation, client endpointing for commands.

# client_endpointing.py
import asyncio

SILENCE_THRESHOLD_MS = 800
MIN_UTTERANCE_MS = 500

class Endpointer:
    def __init__(self, sender):
        self.sender = sender
        self.last_speech_time = None
        self.utterance_started = False
        self._task = None
    
    def on_vad_result(self, is_speech: bool):
        now = asyncio.get_event_loop().time() * 1000
        if is_speech:
            self.last_speech_time = now
            if not self.utterance_started:
                self.utterance_started = True
        else:
            if self.utterance_started and self.last_speech_time:
                if now - self.last_speech_time > SILENCE_THRESHOLD_MS:
                    if now - self.utterance_start_time > MIN_UTTERANCE_MS:
                        self.finalize()
    
    def finalize(self):
        self.sender.send_end_of_stream()
        self.utterance_started = False
        self.last_speech_time = None

Reconnection and session recovery

Networks fail. Mobile clients switch from Wi-Fi to cellular. Your stream must survive.

Strategy: assign a session_id on first config message. On reconnect, send session_id + last acknowledged message_id. Provider resumes from that point if it has buffered state (not all do).

If the provider doesn’t support resume, you have two options:

1. Restart from scratch — acceptable for short utterances
2. Client-side buffer: keep last 30 s of audio locally, replay on reconnect

# reconnect_logic.py
import asyncio
import uuid

class ResilientStream:
    def __init__(self, url, config):
        self.url = url
        self.config = config
        self.session_id = str(uuid.uuid4())
        self.sequence = 0
        self.acked_sequence = 0
        self.audio_buffer = bytearray()
        self.MAX_BUFFER = 30 * 16000 * 2  # 30 seconds
        self.ws = None
    
    async def connect(self):
        self.ws = await websockets.connect(self.url)
        await self.ws.send(json.dumps({
            "type": "config",
            "session_id": self.session_id,
            "resume_from": self.acked_sequence,
            **self.config
        }))
    
    async def send_audio(self, chunk: bytes):
        self.audio_buffer.extend(chunk)
        if len(self.audio_buffer) > self.MAX_BUFFER:
            # drop oldest, keep sequence aligned
            drop = len(self.audio_buffer) - self.MAX_BUFFER
            self.audio_buffer = self.audio_buffer[drop:]
            self.acked_sequence += drop // FRAME_BYTES
        
        await self.ws.send(json.dumps({
            "type": "audio",
            "sequence": self.sequence,
            "data": base64.b64encode(chunk).decode()
        }))
        self.sequence += 1
    
    async def handle_ack(self, ack_seq: int):
        self.acked_sequence = ack_seq
        # trim acknowledged audio from buffer
        acked_bytes = (ack_seq - self.acked_sequence) * FRAME_BYTES
        if acked_bytes > 0:
            self.audio_buffer = self.audio_buffer[acked_bytes:]
    
    async def on_disconnect(self):
        await asyncio.sleep(1)  # backoff
        await self.connect()
        # replay unacknowledged audio
        unacked_frames = (self.sequence - self.acked_sequence)
        if unacked_frames > 0:
            start = -(unacked_frames * FRAME_BYTES)
            await self.send_audio(bytes(self.audio_buffer[start:]))

Pitfall: Replaying audio duplicates transcripts if the provider already processed those frames. Only replay if you’re certain the provider didn’t persist state. Most cloud providers don’t — design for at-least-once delivery and dedup on the client using sequence numbers.

Audio format negotiation

Send 16 kHz, 16-bit PCM, mono. It’s the universal lowest common denominator. Resample on the client if the microphone delivers 44.1 kHz or 48 kHz.

// resample in AudioWorklet
class ResampleProcessor extends AudioWorkletProcessor {
  constructor(options) {
    super();
    this.inputRate = options.processorOptions.inputSampleRate;
    this.outputRate = 16000;
    this.ratio = this.inputRate / this.outputRate;
    this.buffer = new Float32Array(4096);
    this.bufferPos = 0;
  }
  
  process(inputs, outputs) {
    const input = inputs[0][0];
    const output = outputs[0][0];
    
    for (let i = 0; i < input.length; i++) {
      this.buffer[this.bufferPos++] = input[i];
      if (this.bufferPos >= this.buffer.length) this.flush();
    }
    
    let outIdx = 0;
    while (outIdx < output.length) {
      const srcIdx = outIdx * this.ratio;
      const idx = Math.floor(srcIdx);
      const frac = srcIdx - idx;
      if (idx + 1 < this.bufferPos) {
        output[outIdx] = this.buffer[idx] * (1 - frac) + this.buffer[idx + 1] * frac;
      } else {
        break;
      }
      outIdx++;
    }
    return true;
  }
}

Don’t send Opus, MP3, or FLAC unless the provider explicitly documents support and you’ve verified it reduces latency. Transcoding adds CPU and jitter.

Metrics you need in production

Instrument these from day one:

# metrics.py
from dataclasses import dataclass, field
import time

@dataclass
class StreamMetrics:
    session_id: str
    connect_time: float = 0
    first_audio_sent: float = 0
    first_interim_received: float = 0
    first_final_received: float = 0
    reconnects: int = 0
    buffer_overruns: int = 0
    bytes_sent: int = 0
    bytes_received: int = 0
    
    def latency_first_interim(self) -> float:
        if self.first_interim_received and self.first_audio_sent:
            return (self.first_interim_received - self.first_audio_sent) * 1000
        return -1
    
    def latency_e2e(self) -> float:
        if self.first_final_received and self.first_audio_sent:
            return (self.first_final_received - self.first_audio_sent) * 1000
        return -1

Export to Prometheus/OpenTelemetry. Correlate with provider status pages — when latency spikes, you’ll know if it’s you or them.

Common failure modes

Microphone permission denied: Handle NotAllowedError gracefully. Show a persistent UI banner, don’t just log.

Autosuspend on mobile: Browsers throttle background tabs. Use AudioContext with latencyHint: 'interactive' and keep a silent oscillator running to prevent suspend.

const ctx = new AudioContext({ latencyHint: 'interactive' });
const oscillator = ctx.createOscillator();
oscillator.connect(ctx.destination);
oscillator.start();
// inaudible, keeps context alive

Provider rate limits: Implement exponential backoff with jitter. Don’t hammer a 429. If you’re routing across multiple providers, fail over on 429/503/504 — this is where a gateway that handles automatic fallback across 240+ models earns its keep.

Clock drift: Audio timestamps from the microphone and the provider’s word timestamps use different clocks. Don’t try to align them. Use relative offsets only.

Partial UTF-8 in text frames: WebSocket text frames must be valid UTF-8. If you send JSON with base64 audio, you’re fine. If you send binary frames for audio and text frames for JSON, ensure the text frames are complete — don’t split a JSON object across frames.

Testing checklist

1. Network simulation: tc qdisc add dev lo root netem delay 100ms 20ms loss 1% — test reconnection, buffer behavior, latency percentiles.
2. Long session: Run 4+ hours. Watch for memory leaks in audio buffers, WebSocket frame queues, metric cardinality explosion.
3. Overlapping speakers: Feed diarization test sets. Verify interim results don’t flicker wildly when speakers alternate.
4. Silence handling: 30 s silence, then speech. Verify pre-speech buffer captures the first word.
5. Codec mismatch: Record at 48 kHz, verify resampler doesn’t alias or drop samples.
6. Provider failover: Kill the primary provider mid-stream. Verify failover completes within 2 s with no transcript loss.

Deploy considerations

Run the WebSocket termination layer stateless. Scale horizontally behind a load balancer with sticky sessions (or route by session_id header). Keep TLS termination at the edge — don’t decrypt inside the application unless you need to inspect payloads.

If you’re running the model yourself (Whisper.cpp, faster-whisper, NeMo), batch inference requests from multiple streams. GPU utilization jumps from 15% to 85% with batch size 8–16. The tradeoff: added latency equal to batch wait time. Tune for your SLA.

# docker-compose.yml snippet for model server
services:
  asr:
    image: ghcr.io/yourorg/asr-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    environment:
      - BATCH_SIZE=8
      - BATCH_TIMEOUT_MS=50
      - MODEL=large-v3
    ports:
      - "50051:50051"  # gRPC

Real-time speech-to-text streaming is a systems problem disguised as an ML problem. The model is a black box you call; the engineering is everything around it — transport, buffering, backpressure, reconnection, observability. Get the plumbing right and the model quality becomes the only variable you can’t control.