What is text-to-speech and how neural TTS works

Text-to-speech (TTS) is the task of converting written text into natural-sounding audio waveforms. Modern neural TTS systems replace the concatenative and parametric pipelines of the past with end-to-end deep learning models that predict acoustic features — typically mel-spectrograms — and synthesize raw audio through a neural vocoder. The result is controllable, expressive speech that can adapt to speaker identity, language, and prosody without hand-crafted rules.

How neural TTS works

Neural TTS decomposes into two main stages: an acoustic model that maps text to spectral representations, and a vocoder that turns those representations into waveform samples. Both stages are differentiable and trained on paired (text, audio) data.

Text frontend and normalization

Raw text never feeds directly into the acoustic model. A text frontend first normalizes input: expanding abbreviations (“Dr.” → “doctor”), converting numbers (“2024” → “twenty twenty-four”), handling currency, dates, and homograph disambiguation (“read” present vs. past tense). Grapheme-to-phoneme (G2P) conversion then maps normalized text to phoneme sequences, often with stress and boundary markers.

# Simplified frontend pipeline
def text_to_phonemes(text: str, language: str = "en-us") -> list[int]:
    normalized = normalize(text, language)      # "I have $5.50" → "I have five dollars fifty cents"
    phonemes = g2p(normalized, language)        # → ["aɪ", "hæv", "faɪv", "dɒlɚz", "fɪfti", "sɛnts"]
    return [phoneme_to_id[p] for p in phonemes]  # → [42, 17, 89, 103, 55, 28]

Most production systems use a hybrid G2P: a dictionary lookup for known words plus a neural fallback (typically a small transformer or RNN) for out-of-vocabulary terms. The output is a sequence of token IDs — phonemes, characters, or byte-pair encoded subwords — ready for the acoustic model.

Acoustic model: text to mel-spectrogram

The acoustic model predicts a mel-spectrogram (or linear spectrogram) frame by frame. Early neural approaches like Tacotron 2 used an encoder-decoder with attention: a bidirectional LSTM or transformer encoder processes phonemes, an autoregressive decoder generates mel frames conditioned on previous frames, and a location-sensitive attention mechanism aligns text to audio.

Modern systems favor non-autoregressive (parallel) architectures for speed. FastSpeech 2 and its variants replace the autoregressive decoder with a feed-forward transformer that predicts the entire mel sequence in one pass, guided by a duration predictor that expands phoneme embeddings to frame-level resolution. A variance predictor adds pitch and energy contours.

# FastSpeech 2 style forward pass (simplified)
class FastSpeech2(nn.Module):
    def forward(self, phoneme_ids: Tensor, speaker_id: Tensor = None) -> Tensor:
        x = self.phoneme_embedding(phoneme_ids)           # [B, T_phon, D]
        x = self.encoder(x)                               # [B, T_phon, D]
        
        # Duration predictor: phoneme-level → frame-level expansion
        log_durations = self.duration_predictor(x)        # [B, T_phon]
        durations = torch.exp(log_durations).round().long()
        x = expand_by_duration(x, durations)              # [B, T_frame, D]
        
        # Variance predictors add prosody
        pitch = self.pitch_predictor(x)                   # [B, T_frame, 1]
        energy = self.energy_predictor(x)                 # [B, T_frame, 1]
        x = x + self.pitch_embedding(pitch) + self.energy_embedding(energy)
        
        mel = self.decoder(x)                             # [B, T_frame, n_mels]
        return mel

Training uses a combination of L1/L2 mel-spectrogram loss, duration loss (against forced-aligner targets), and adversarial or feature-matching losses from a discriminator. Speaker conditioning — via embedding lookup or global style tokens — enables multi-speaker models from a single checkpoint.

Vocoder: mel to waveform

The vocoder reconstructs time-domain audio from the predicted mel-spectrogram. Autoregressive WaveNet and WaveRNN produce high fidelity but are slow. Flow-based models (Glow-TTS, WaveGlow) and GAN-based vocoders (HiFi-GAN, BigVGAN, Vocos) generate in parallel with near-real-time factors on GPU and increasingly on CPU.

HiFi-GAN remains a production workhorse: a multi-scale, multi-period discriminator adversarially trains a generator composed of transposed convolutions with residual blocks. The generator takes mel-spectrograms as input and outputs 22.05 kHz or 24 kHz audio.

# HiFi-GAN generator inference (conceptual)
class HiFiGANGenerator(nn.Module):
    def forward(self, mel: Tensor) -> Tensor:
        # mel: [B, n_mels, T_frame]
        x = self.conv_pre(mel)                            # [B, C, T_frame]
        for upsample_block in self.upsample_blocks:
            x = upsample_block(x)                         # progressive upsampling
        x = self.conv_post(x)                             # [B, 1, T_audio]
        return torch.tanh(x).squeeze(1)                   # [B, T_audio]

Vocoder choice trades off quality, speed, and footprint. For server-side batch inference, HiFi-GAN or BigVGAN on GPU is standard. For on-device or edge deployment, quantized Vocos or streamable variants of HiFi-GAN run at >50× real-time on modern mobile SoCs.

Why neural TTS matters for engineers

Neural TTS shifts the integration surface from “pick a voice from a vendor catalog” to “configure a model pipeline.” This creates both leverage and operational complexity.

Controllability. Prosody, speaking rate, pitch, and speaker identity become tensor inputs rather than SSML tags. You can interpolate between speakers, clone a voice from a 10-second reference (zero-shot TTS), or steer emotion via style embeddings — all at inference time.

Latency profile. Autoregressive acoustic models (Tacotron 2) have O(T) sequential steps; non-autoregressive models (FastSpeech 2, VITS) are O(1) parallel passes. Vocoder choice dominates end-to-end latency. A typical server stack: FastSpeech 2 + HiFi-GAN on A100 yields ~15 ms per second of audio (65× real-time). Streaming variants can emit first audio chunk in <100 ms.

Data efficiency. Modern architectures train on 10–100 hours of single-speaker data for high quality. Multi-speaker models scale to thousands of speakers with 1–5 minutes each using speaker embeddings. Fine-tuning a pre-trained backbone to a new voice takes GPU-hours, not GPU-weeks.

Artifacts and failure modes. Neural TTS fails differently than concatenative systems. Common artifacts: mispronounced rare words (G2P gaps), prosody collapse on long sentences (attention drift), metallic buzzing (vocoder spectral holes), and speaker leakage in multi-speaker models. Monitoring requires audio-quality metrics (MOS prediction, PESQ, STOI) alongside text-level checks.

Concrete example: building a streaming TTS endpoint

Consider a service that streams audio to a WebSocket client as it’s generated. The acoustic model must support incremental decoding, and the vocoder must be causal (no future context).

# Streaming TTS with VITS (non-autoregressive, flow-based)
# VITS combines acoustic model + vocoder in one end-to-end model

class StreamingTTS:
    def __init__(self, model_path: str, device: str = "cuda"):
        self.model = VITS.load(model_path).to(device).eval()
        self.sample_rate = 22050
        self.chunk_size = 1024  # samples per yield (~46 ms at 22.05 kHz)
    
    async def synthesize_stream(self, text: str, speaker_id: int = 0) -> AsyncGenerator[bytes, None]:
        phonemes = text_to_phonemes(text)
        phoneme_tensor = torch.tensor([phonemes], device=self.model.device)
        speaker_tensor = torch.tensor([speaker_id], device=self.model.device)
        
        # VITS generates waveform directly, but we chunk for streaming
        with torch.inference_mode():
            audio = self.model.infer(phoneme_tensor, speaker_tensor)  # [1, T_audio]
        
        audio = audio.squeeze(0).cpu().numpy().astype(np.float32)
        
        # Yield chunks
        for i in range(0, len(audio), self.chunk_size):
            chunk = audio[i:i + self.chunk_size]
            # Convert float32 [-1, 1] → int16 bytes
            yield (chunk * 32767).astype(np.int16).tobytes()
            await asyncio.sleep(0)  # yield control

Key production concerns this snippet omits: request queuing with priority, model warm-up, GPU memory pooling, client disconnect handling, and audio format negotiation (Opus, MP3, raw PCM). The n4n.ai gateway handles several of these at the infrastructure layer — model routing, fallback, and usage metering — so application code stays focused on the TTS logic.

Common misconceptions

“Neural TTS solves pronunciation.” It doesn’t. G2P errors persist on proper nouns, domain terminology, and heteronyms. Production systems layer custom lexicons, phoneme-level override APIs, and sometimes a small pronunciation correction model fine-tuned on domain data.

“One model fits all languages.” Multilingual models (XTTS, VALL-E X, Bark) share representations across languages but still exhibit accent bleed and code-switching artifacts. Per-language fine-tuning or language-specific heads remain standard for quality-critical deployments.

“SSML is the right control interface.” SSML is verbose, inconsistently implemented, and maps poorly to neural controls (speaker embeddings, style vectors). A JSON API with explicit fields — speaker_id, speed, pitch_shift, style_vector — is more maintainable and versionable.

{
  "text": "The deployment completed successfully.",
  "speaker": "en-us-female-1",
  "prosody": {"rate": 1.1, "pitch": -2},
  "style": "calm",
  "format": "opus",
  "sample_rate": 24000
}

“Real-time factor (RTF) tells the whole latency story.” RTF measures compute time per audio second. It ignores queue time, model loading, text frontend latency, and network serialization. End-to-end latency at p99 is the metric that correlates with user experience.

“You need massive data for a custom voice.” Zero-shot and few-shot TTS (VALL-E, VoiceBox, OpenVoice) clone speakers from 3–30 seconds of reference audio. Quality varies with reference clarity and acoustic match to training data, but the data barrier has effectively fallen for many use cases.

Architecture decisions checklist

When evaluating or designing a TTS stack, decide explicitly on:

Evaluation you can automate

Subjective MOS (Mean Opinion Score) is the gold standard but expensive. Automated proxies let you gate merges:

• Pronunciation accuracy: forced-align generated audio against reference phonemes; compute phone error rate.
• Speaker similarity: cosine similarity between speaker embeddings (ECAPA-TDNN, WavLM) of generated vs. reference audio.
• Prosody naturalness: pitch/energy contour correlation with ground truth; F0 RMSE on voiced frames.
• Artifact detection: spectral hole density (vocoder buzz), repetition detection (attention loops), silence insertion rate.

Run these on a held-out test set (500+ utterances covering short/long, common/rare words, multiple speakers) on every model checkpoint.

Where the field is moving

• Diffusion and flow-matching vocoders (VoiceBox, NaturalSpeech 3) surpass GANs on naturalness and enable zero-shot editing (inpainting, continuation).
• Language models for TTS (VALL-E, Spear-TTS) treat audio as discrete tokens from a neural codec (EnCodec, DAC), enabling LLM-style scaling laws and in-context learning.
• Streaming-first architectures (StreamSpeech, CosyVoice) unify simultaneous translation and TTS, emitting audio tokens as source text arrives.
• Expressive control via text prompts (“speak excitedly but quietly”) replaces categorical style labels, using CLAP or LLM embeddings to condition generation.

Neural TTS has moved from research curiosity to commodity infrastructure. The engineering challenge is no longer “can we generate speech?” but “can we serve the right voice, with the right prosody, at the right latency, within the right cost envelope, while detecting regressions before users do?”