What is speech-to-text AI and how it transcribes audio

What is speech-to-text AI is the task of mapping an acoustic waveform to a sequence of text tokens. Modern systems treat this as a sequence-to-sequence problem: an acoustic encoder compresses raw audio into latent representations, and a language-model-informed decoder emits tokens conditioned on both the acoustic context and prior text history. The result is a transcript that preserves linguistic content while discarding speaker identity, background noise, and channel artifacts — unless you explicitly ask to keep them.

How the pipeline works

Front-end: feature extraction

Raw PCM at 16 kHz (or 8 kHz for telephony) enters a short-time Fourier transform or filter-bank layer producing log-mel spectrograms — typically 80 channels, 25 ms windows, 10 ms stride. SpecAugment (time warping, frequency masking, time masking) applies on-the-fly during training to improve robustness. Some architectures (Whisper, SeamlessM4T) skip hand-crafted features entirely and learn a convolutional front-end from raw waveforms.

# Typical feature config for a 16 kHz model
N_MELS = 80
N_FFT = 400          # 25 ms at 16 kHz
HOP_LENGTH = 160     # 10 ms stride
WIN_LENGTH = 400

Encoder: acoustic modeling

The encoder consumes the mel sequence and outputs a hidden state per frame (or per subsampled frame). Dominant architectures:

Subsampling reduces frame rate from 100 Hz to ~25 Hz, cutting compute and memory for the decoder. Encoder output dimension is usually 512–1024.

Decoder: token generation

Two families dominate production systems:

CTC (Connectionist Temporal Classification) emits a token per subsampled frame independently, then collapses repeats and blanks. Fast, parallelizable, but no explicit language model — it leans on the encoder’s implicit LM. Used alone for low-latency streaming or as a first-pass in hybrid systems.

Attention-based sequence-to-sequence (AED / Transducer) conditions each output token on the full encoder output and previously emitted tokens. An RNN-T (RNN Transducer) adds a prediction network (joiner) that fuses encoder and predictor states per step, enabling streaming with bounded latency. Whisper uses a standard Transformer decoder with cross-attention to the encoder.

# Pseudocode: RNN-T joint step
def rnnt_step(enc_out_t, pred_state, joiner, predictor, vocab):
    # enc_out_t: [D] encoder output at time t
    # pred_state: predictor hidden state
    pred_out, next_pred_state = predictor(pred_state)  # [D]
    joint = joiner(enc_out_t, pred_out)                # [V]
    logits = joint.log_softmax(dim=-1)
    return logits, next_pred_state

Language model fusion

Production pipelines almost always fuse an external LM at inference:

• Shallow fusion: interpolate LM log-probs into decoder logits at each step (log P(y|x) + λ log P_LM(y)).
• Deep fusion: concatenate LM hidden states into the decoder (rare in serving due to latency).
• Rescoring: run first-pass (CTC or RNN-T) to generate N-best hypotheses, then rescore with a large Transformer LM.

A 4-gram KenLM or a pruned Transformer LM (100–300M params) is typical for on-device or low-latency server inference. Larger LMs (1B+) run in a second-pass reranker.

Tokenization

Byte-pair encoding (BPE) or Unigram subword vocabularies of 1k–10k tokens are standard. Whisper uses a 51865-token multilingual vocabulary with language tags (<|en|>, <|zh|>) and task tokens (<|transcribe|>, <|translate|>). Character-level output avoids OOV but increases sequence length 3–4×.

Streaming vs batch: the latency-accuracy trade-off

Streaming encoders use causal attention masks and chunked convolutions. The decoder emits tokens as soon as the joiner confidence exceeds a threshold (RNN-T) or after a fixed lookahead (Attention). Endpointing — detecting speech end — couples with the decoder: a silence detector or the model’s own <eos> probability triggers finalization.

# Chunked encoder forward (simplified)
def streaming_encode(chunk, cache, chunk_size=1600, lookahead=640):
    # chunk: [B, chunk_size] raw audio
    # cache: previous conv states, attention keys/values
    feats = frontend(chunk)                    # [B, T_chunk, 80]
    feats = torch.cat([cache['feat'], feats], dim=1)
    enc_out, new_cache = encoder(feats, cache)
    # Return only frames corresponding to current chunk (minus lookahead)
    return enc_out[:, :T_chunk], new_cache

Why it matters for system design

Accuracy metrics that matter

Word Error Rate (WER) is the standard: WER = (S + D + I) / N where S/D/I are substitutions, deletions, insertions, N is reference word count. But WER hides operational reality:

• Entity error rate: names, numbers, addresses — critical for voice assistants.
• Latency percentiles: p50, p95, p99 end-to-end (audio in → first token / final transcript).
• Real-time factor (RTF): compute time / audio duration. RTF < 0.3 enables 3× faster-than-real-time batch; RTF < 1.0 required for streaming.
• Failure modes: hallucination rate (inserting words during silence), language confusion, speaker diarization errors.

Deployment architecture

A typical serving stack:

Client → [Load Balancer] → [Preprocessing: VAD, resample, normalize]
                              ↓
                    [Model Server: Triton / vLLM / custom]
                              ↓
                    [Postprocessing: ITN, punctuation, diarization]
                              ↓
Client ← [WebSocket / gRPC / HTTP streaming]

• VAD (Voice Activity Detection) runs before the ASR model to drop silence and segment utterances. Silero VAD (1.5 MB, ONNX) is common.
• ITN (Inverse Text Normalization) converts spoken form (“twenty twenty three”) to written (“2023”). Rule-based (NVIDIA NeMo ITN) or neural.
• Punctuation/capitalization often a separate BERT-style tagger running on ASR output.
• Diarization (who spoke when) runs in parallel or post-hoc: pyannote.audio, NVIDIA MSDD, or integrated EEND models.

Scaling considerations

• Batching: Dynamic batching pads variable-length audio to the longest in the batch. For streaming, batch size = concurrent sessions. GPU utilization drops sharply below 8–16 concurrent streams on A100/H100.
• Model variants: Distilled small models (Whisper tiny/base, 39M/74M params) run 10–20× faster than large-v3 (1.5B) with 10–15% relative WER degradation. Quantization (INT8, INT4 via GPTQ/AWQ) recovers another 2–3× with minimal quality loss.
• Multi-model routing: Route short utterances to a small streaming model; long-form to a large batch model. This is where a gateway that honors client routing directives and forwards provider cache-control hints reduces operational complexity — you declare the policy once instead of baking it into every client.

Concrete example: transcribing a 30-minute podcast

Input: 30 min mono WAV, 16 kHz, two speakers, music intro/outro.

Pipeline:

1. VAD segments into ~120 speech chunks (avg 15 s), drops music.
2. Diarization assigns speaker labels per chunk (Speaker A / B).
3. Batch ASR (Whisper large-v3, fp16, batch size 16 on A100) processes chunks in parallel. RTF ~0.08 → 2.4 min wall time.
4. ITN + punctuation restores numbers, dates, sentence boundaries.
5. Alignment (WhisperX or gentle) forces word-level timestamps using the transcript as supervision.
6. Merge by speaker, output WebVTT/SRT/JSON.

{
  "segments": [
    {
      "start": 12.3,
      "end": 18.7,
      "speaker": "A",
      "text": "Welcome back to the show. Today we're talking about inference optimization."
    },
    {
      "start": 19.1,
      "end": 24.5,
      "speaker": "B",
      "text": "Right. And specifically how to serve large models without going broke."
    }
  ],
  "language": "en",
  "duration": 1842.3
}

Cost envelope (rough, self-hosted A100 80GB spot): ~$0.02/minute audio for large model, ~$0.003/min for distilled small model. Cloud APIs range $0.006–$0.024/min depending on tier and volume.

Common misconceptions

“Whisper is the only model you need”

Whisper generalizes well across domains and languages, but it has blind spots: heavy accents, code-switching mid-utterance, domain-specific terminology (medical, legal, programming), and noisy telephony audio. Fine-tuning on 50–100 hours of in-domain data typically yields 15–30% relative WER reduction. Specialized models (Conformer-Transducer trained on 100k+ hours of call-center audio) still beat Whisper on their target domain.

“Streaming is just batch with smaller chunks”

Chunked attention without causal masking leaks future context, artificially lowering WER. True streaming requires causal encoder attention, frame-synchronous decoding, and endpointing that doesn’t wait for silence. Latency budgets of 300 ms end-to-end leave ~150 ms for model inference — ruling out large Transformers without distillation or speculative decoding.

“WER tells the whole story”

A 5% WER on clean read speech ≠ 5% WER on a noisy conference call with overlapping speakers. Always evaluate on your data distribution. Build a regression set: 2–4 hours spanning your acoustic conditions, speakers, and vocabulary. Track WER, entity error rate, and hallucination rate per release.

“You need GPUs for inference”

INT8-quantized Conformer-Transducer (30M params) runs at RTF 0.15 on a modern x86 CPU (AVX2) — 6× real-time. ARM NEON builds achieve similar. For batch workloads, CPU inference is often cheaper per audio-hour than GPU if you have spare cores. The break-even depends on your cloud CPU vs GPU pricing and concurrency needs.

“Diarization + ASR = speaker-attributed transcript”

Diarization errors (missed speaker turns, false splits) compound with ASR errors. Joint models (EEND, TS-VAD) reduce this but add complexity. A pragmatic approach: run diarization first, then constrain ASR decoding per segment with speaker-adapted LMs, then realign. Accept that 5–10% of speaker labels will be wrong in noisy conditions.

Integration checklist for engineers

• Define your latency budget (first-token, final) and accuracy targets per use case.
• Collect a representative eval set before model selection.
• Choose model size: start with distilled small + LM fusion; scale up only if eval demands it.
• Implement VAD + endpointing before the model — don’t feed silence.
• Add ITN, punctuation, and diarization as separate, replaceable stages.
• Instrument RTF, p50/p95/p99 latency, GPU/CPU utilization, and error rates in production.
• Plan for model updates: A/B test new checkpoints against your regression set.
• Consider a gateway layer if you route across multiple models/providers — it centralizes fallback, metering, and routing logic without duplicating it in every service.

Speech-to-text is a solved problem for clean, single-speaker, high-resource languages. The engineering challenge is the long tail: noise, accents, code-switching, domain vocabulary, latency constraints, and cost at scale. Treat the model as one component in a pipeline you can measure, swap, and optimize independently.