How Whisper compares to newer speech-to-text models

Whisper vs newer speech-to-text models is a comparison every voice-enabled product team faces once they move past prototyping. OpenAI’s Whisper changed the landscape by making high-quality, multilingual transcription openly available, but the ecosystem has moved fast since its 2022 release. Commercial APIs now offer lower latency, speaker diarization, word-level timestamps, and streaming support that the base Whisper model lacks. Self-hosted forks like Whisper.cpp and WhisperX close some gaps but introduce operational complexity. This breakdown covers the concrete trade-offs so you can pick the right tool without guesswork.

Model landscape and positioning

Whisper (the original OpenAI release) comes in five sizes — tiny, base, small, medium, large — with the large-v3 model at 1.5B parameters. It was trained on 680k hours of weakly supervised multilingual data. The model is autoregressive: it generates tokens sequentially, which fundamentally limits throughput and makes true streaming difficult without chunking hacks.

Newer entrants take different architectural approaches. Deepgram Nova-3 and AssemblyAI Universal-2 use encoder-decoder transformers with connectionist temporal classification (CTC) or transducer heads, enabling parallel decoding and native streaming. Gladia’s Whisper-Zero distills Whisper into a non-autoregressive architecture for speed. Google Chirp 2 and Azure AI Speech use proprietary conformer-transducer hybrids trained on millions of supervised hours. Rev AI and Speechmatics lean on large-scale semi-supervised pipelines with heavy emphasis on proper noun accuracy.

The practical distinction: Whisper is a model you run. The newer commercial offerings are managed services with SLAs, streaming APIs, and feature bundles (diarization, summarization, PII redaction) built in. Self-hosted forks sit in between — you operate the infrastructure but gain features like word-level timestamps and diarization via WhisperX’s alignment pipeline.

Capabilities comparison

Whisper’s 30-second context window is a hard constraint. Long-form transcription requires chunking with overlap, which creates boundary artifacts and duplicate words. WhisperX mitigates this with voice activity detection (VAD) segmentation and forced alignment, but you’re still stitching segments. The commercial APIs handle arbitrarily long streams natively — a meaningful operational difference for call centers, podcasts, or meeting recorders.

Custom vocabulary is another sharp edge. Whisper only accepts a prompt prefix (the “initial prompt” parameter), which biases the decoder but doesn’t guarantee recognition of rare proper nouns. Deepgram, AssemblyAI, and Azure let you submit phrase lists or train lightweight custom models that actually shift the beam search. If your domain has specialized terminology — medical codes, legal citations, product names — this matters.

Price and cost model

Whisper is free to run, but GPU time isn’t. On an A10G (24GB VRAM), large-v3 processes ~40-50x real-time audio with batching. That’s roughly $0.10-0.15/hour of audio on spot instances, plus engineering time for the serving stack, queue management, autoscaling, and monitoring. Whisper.cpp on CPU (AVX2/NEON) runs ~3-5x real-time on modern x86/ARM — viable for batch workloads, not for latency-sensitive paths.

Commercial APIs price per minute of audio:

• Deepgram Nova-3: $0.0043/min (pre-recorded), $0.0059/min (streaming)
• AssemblyAI Universal-2: $0.0037/min (pre-recorded), $0.0050/min (streaming)
• Gladia: $0.0060/min (includes diarization, translation)
• Azure AI Speech: $0.0165/min (standard), $0.024/min (custom model)
• Google Chirp 2: $0.012/min (standard), $0.024/min (with data logging opt-out)
• Rev AI: $0.020/min (async), $0.035/min (streaming)
• Speechmatics: ~$0.008/min (volume tiers available)

At 10,000 hours/month, Deepgram runs ~$2,600 vs. ~$600-900 for self-hosted A10Gs (2-3 GPUs) — but the self-hosted number excludes SRE overhead, which often tips the scale back toward managed services for teams under 10 engineers. Whisper vs newer speech-to-text models often comes down to whether you treat inference as a core competency or a utility bill.

Latency and throughput

Autoregressive decoding is Whisper’s bottleneck. Each token attends to all previous tokens, so generation time scales quadratically with sequence length. Batched inference helps throughput but hurts latency — you wait for the slowest sequence in the batch. Typical numbers on A10G:

• Whisper large-v3 (batched, 30s chunks): ~200-400ms per 30s chunk → ~7-13ms/second of audio (throughput), but end-to-end latency per chunk is 200-400ms + queue time
• Whisper.cpp (CPU, single thread): ~3-5x real-time → 200-330ms latency per second of audio
• WhisperX (with VAD + alignment): Adds 100-300ms for alignment pass

Streaming commercial APIs:

• Deepgram Nova-3 streaming: ~150-300ms end-to-end (first token ~100ms)
• AssemblyAI Universal-2 streaming: ~200-400ms end-to-end
• Gladia streaming: ~400-600ms end-to-end
• Azure/Google streaming: ~150-300ms end-to-end

If you need partial results while the user speaks (live captions, voice assistants), Whisper’s chunked approach introduces 2-5 second delays minimum. The commercial streaming APIs deliver partials every 100-500ms. For batch transcription (podcasts, call recordings), Whisper’s throughput is competitive; for real-time, it’s not.

Ergonomics and developer experience

Whisper’s API surface is minimal: model.transcribe(audio, language=None, task="transcribe", initial_prompt=None). You handle chunking, VAD, speaker diarization, timestamp alignment, and output formatting yourself. WhisperX wraps this into a pipeline:

import whisperx

device = "cuda"
model = whisperx.load_model("large-v3", device, compute_type="float16")
audio = whisperx.load_audio("meeting.wav")

# Transcribe with VAD segmentation
result = model.transcribe(audio, batch_size=16, language="en")

# Align for word-level timestamps
model_a, metadata = whisperx.load_align_model(language_code="en", device=device)
result = whisperx.align(result["segments"], model_a, metadata, audio, device)

# Diarize
diarize_model = whisperx.DiarizationPipeline(use_auth_token=HF_TOKEN, device=device)
diarize_segments = diarize_model(audio)
result = whisperx.assign_word_speakers(diarize_segments, result)

Three models loaded, two pipeline stages, manual speaker assignment. It works but it’s brittle — version mismatches between whisperx, pyannote, and transformers break frequently.

Commercial APIs are REST or WebSocket endpoints with JSON payloads:

curl -X POST "https://api.deepgram.com/v1/listen?model=nova-3&diarize=true&punctuate=true&smart_format=true" \
  -H "Authorization: Token $DEEPGRAM_KEY" \
  -H "Content-Type: audio/wav" \
  --data-binary @meeting.wav

{
  "results": {
    "channels": [{
      "alternatives": [{
        "transcript": "Hello, this is a test.",
        "confidence": 0.98,
        "words": [
          {"word": "Hello", "start": 0.12, "end": 0.45, "confidence": 0.99, "speaker": 0},
          {"word": "this", "start": 0.51, "end": 0.68, "confidence": 0.97, "speaker": 0}
        ]
      }]
    }]
  }
}

WebSocket streaming is similarly straightforward — send audio chunks, receive partial transcripts with is_final flags. SDKs exist for Python, Node, Go, Rust. The ergonomics gap is real: commercial APIs remove entire categories of glue code.

Ecosystem and integrations

Whisper lives in the Hugging Face / PyTorch ecosystem. Integrations exist for LangChain, LlamaIndex, FastAPI, Ray Serve, Triton, vLLM (experimental). Community forks add features: faster-whisper (CTranslate2 backend, 4x speedup), whisper.cpp (GGML, runs everywhere), whisper-jax (TPU), insanely-fast-whisper (flash attention + batching). But each fork has its own quirks, maintenance cadence, and compatibility matrix.

Commercial APIs integrate with the platforms you already use:

• Deepgram: Twilio, Vonage, Plivo, LiveKit, Daily, Symbl.ai, LangChain callbacks
• AssemblyAI: Webhooks, LeMUR (LLM on transcripts), Make/Zapier, Retool
• Azure: Cognitive Services SDK, Speech Studio no-code customization, AKS deployment
• Google: Vertex AI, Dialogflow CX, Contact Center AI
• Gladia: LiveKit, Daily, custom webhooks, translation + summarization endpoints

If your stack includes Twilio Media Streams or LiveKit for real-time voice, Deepgram and Gladia have native integrations that handle the audio format negotiation and packetization. With Whisper, you build that plumbing.

Operational limits and failure modes

Self-hosted Whisper fails in predictable ways:

• OOM on long audio: The 30s chunk workaround helps, but batch size × sequence length × model size hits VRAM limits fast. faster-whisper with compute_type="int8" reduces VRAM ~50% with <1% WER regression.
• Queue buildup under burst traffic: Autoregressive decoding doesn’t parallelize across time steps. You need request batching (vLLM-style continuous batching doesn’t apply cleanly here) or multiple model replicas behind a load balancer.
• Model drift: Whisper checkpoints are static. If your audio distribution shifts (new accents, domain vocabulary), you can’t fine-tune easily — LoRA on Whisper is experimental and breaks the decoder’s language modeling.
• Dependency hell: whisperx pins torch, torchaudio, pyannote, transformers, accelerate. Upgrading one breaks another. Pin versions in Docker.

Commercial APIs fail differently:

• Rate limits: Deepgram defaults to 300 concurrent streams; AssemblyAI to 100. Request increases require support tickets.
• Data residency: EU/US region locking matters for regulated industries. Azure and Google offer regional endpoints; Deepgram and AssemblyAI are US-only unless you negotiate enterprise.
• Vendor lock-in: Transcript formats differ. Migration means rewriting parsers and re-evaluating quality on your data.
• Pricing surprises: Streaming minutes cost 1.5-2x async. Diarization, translation, and summarization are often add-ons.

Accuracy on real-world audio

Benchmarks on clean read speech (LibriSpeech, Common Voice) show all top models at 3-5% WER. The divergence appears on:

• Telephony audio (8kHz, codecs, noise): Whisper degrades sharply below 16kHz. Commercial models trained on call center data (Deepgram, AssemblyAI, Rev) hold 8-12% WER where Whisper hits 20%+.
• Overlapping speech: Whisper transcribes the dominant speaker. WhisperX + pyannote separates speakers but assigns words post-hoc. Native diarization models (Deepgram, AssemblyAI) handle overlap better because speaker embeddings inform the acoustic model jointly.
• Accented speech: Whisper’s multilingual training helps, but low-resource accents still suffer. AssemblyAI Universal-2 and Google Chirp 2 invest heavily in accented English — measurable gains on African, Indian, and Southeast Asian accents.
• Proper nouns: Without custom vocabulary, all models hallucinate. Whisper’s prompt injection helps ~30% of cases. Keyword boosting in commercial APIs helps ~70%.

Run your own eval. Download 50-100 representative clips, transcribe with each candidate, compute WER against human references. The ranking often surprises teams who trust leaderboard numbers.

Which to choose

Choose Whisper (self-hosted) when:

• You have GPU infrastructure and SRE capacity to operate it
• Batch transcription of batch transcription workloads (podcasts, media archives, call recordings)
• You need full data isolation — audio never leaves your VPC
• Your language mix includes low-resource languages poorly covered by commercial APIs
• Cost at scale (>50k hours/month) favors amortized GPU over per-minute pricing

Choose WhisperX / faster-whisper when:

• You need word-level timestamps and speaker diarization but want to self-host
• You can tolerate Python dependency management and occasional pipeline breakage
• Latency requirements are relaxed (async processing acceptable)

Choose Deepgram Nova-3 when:

• You need native streaming with sub-300ms latency for live captions, voice assistants, or real-time analytics
• Telephony audio (8kHz, noise, crosstalk) is your primary input
• You want keyword boosting without training custom models
• You integrate with Twilio, LiveKit, or Daily and want drop-in media stream handling

Choose AssemblyAI Universal-2 when:

• Accented English coverage is critical (global user base, diverse speakers)
• You want LeMUR for LLM-powered summarization, QA, or entity extraction on transcripts
• Webhook-based async workflows fit your architecture better than WebSocket streaming
• You need strong low-resource language support beyond the top 30

Choose Gladia when:

• You need translation + transcription in one API call (99 languages)
• LiveKit or Daily integration is required
• You want a single bill for transcription, diarization, translation, and summarization

Choose Azure AI Speech or Google Chirp 2 when:

• You’re already on Azure/GCP and want unified billing, IAM, and VPC controls
• You need custom model training (acoustic + language model) for domain-specific accuracy
• Data residency in specific regions (EU, GovCloud, etc.) is mandatory
• You need on-premises deployment options (Azure Edge, Google Distributed Cloud)

Choose Rev AI or Speechmatics when:

• You need human-in-the-loop verification workflows (Rev)
• You need the strongest proper noun / entity accuracy for legal, medical, or financial domains (Speechmatics)
• You have enterprise procurement processes that favor established vendors

Practical migration path

If you’re on Whisper today and evaluating a switch:

1. Export 200 representative clips covering your audio conditions, speakers, languages, and domain vocabulary.
2. Run batch eval against 2-3 commercial APIs using their async endpoints. Compute WER, speaker error rate, and latency percentiles.
3. Test streaming with your actual client integration (WebSocket, media stream). Measure time-to-first-token and partial stability.
4. Calculate TCO at your projected volume: GPU hours × cloud rate × 1.5 (ops overhead) vs. API minutes × rate × 1.2 (buffer for streaming premium).
5. Run a shadow period — dual-write transcripts to your store for 2 weeks. Compare downstream task quality (summarization, classification, search) before cutting over.

The model is rarely the blocker. The integration surface, operational model, and failure semantics determine whether a transcription pipeline survives production.