Decoder-only vs encoder-decoder models explained

The decoder-only vs encoder-decoder distinction shapes every decision downstream: how you prompt, what you can condition on, how much compute you burn, and which failure modes keep you up at night. Most engineers only encounter decoder-only models because that’s what the major APIs serve, but encoder-decoder architectures still dominate translation, summarization, and any task where bidirectional context matters. Understanding both lets you pick the right tool instead of forcing everything into a chat completion endpoint.

Architecture fundamentals

Decoder-only models (GPT, LLaMA, Mistral, Claude) stack causal self-attention layers that attend only to prior tokens. The training objective is next-token prediction over massive corpora. At inference, you feed a prompt and the model generates autoregressively, conditioning each new token on everything before it.

Encoder-decoder models (T5, BART, mT5, UL2, Flan-T5) split the stack. The encoder ingests the full input sequence with bidirectional self-attention — every token sees every other token. The decoder then generates autoregressively, attending to its own prior tokens and the full encoder output via cross-attention. The training objective is typically span corruption or sequence-to-sequence denoising.

This structural difference cascades into every practical dimension.

Capabilities comparison

Bidirectional understanding. Encoder-decoder models see the entire input at once during encoding. This matters for tasks where later context clarifies earlier ambiguity — think coreference resolution in long documents, or translation where the target language’s gender agreement depends on a noun three sentences later. Decoder-only models only see left context during generation, though large context windows and in-context learning mitigate this in practice.

Conditional generation control. Encoder-decoder architectures natively support structured conditioning: the encoder output is a fixed representation you can manipulate, inspect, or reuse across multiple decoder passes. Decoder-only models require the entire condition to be in the prompt, which bloats context and prevents reuse without recomputation.

In-context learning. Decoder-only models excel at few-shot adaptation because their training distribution (internet-scale text) contains endless patterns of “examples followed by completion.” Encoder-decoder models can do in-context learning but typically need explicit fine-tuning (Flan-style instruction tuning) to match decoder-only flexibility on arbitrary tasks.

Reasoning and open-ended generation. Decoder-only models dominate here. Their training on diverse, unstructured text builds broad world knowledge and chain-of-thought reasoning capabilities that encoder-decoder models — usually trained on more structured denoising objectives — struggle to match without massive scale.

Training and inference economics

Training compute. For equivalent parameter counts, encoder-decoder models are roughly 1.5–2× more expensive to train. The encoder processes the full sequence bidirectionally (quadratic attention over full length), while decoder-only training uses causal masking that enables efficient causal attention kernels. At 7B+ parameters, this gap compounds.

Inference compute. Decoder-only inference is a single forward pass per generated token with KV caching. Encoder-decoder inference requires one full encoder pass (quadratic in input length) plus one decoder pass per token (with cross-attention to encoder outputs). For short inputs and long outputs, decoder-only wins. For long inputs and short outputs — summarization, classification, extraction — encoder-decoder can be cheaper because the encoder runs once and the decoder emits few tokens.

KV cache memory. Decoder-only KV cache grows with total sequence length (prompt + generation). Encoder-decoder KV cache splits: encoder cache is fixed at input length, decoder cache grows only with output length. For very long prompts with short completions, encoder-decoder uses less peak memory.

Latency and throughput characteristics

First-token latency matters for streaming UX. Decoder-only models start emitting immediately. Encoder-decoder models must finish the encoder pass before the first decoder token — a noticeable pause on long inputs. For batch throughput, decoder-only models saturate GPUs more easily because the causal attention pattern maps cleanly to flash attention kernels. Encoder bidirectional attention is less kernel-friendly, though flash attention v2+ has narrowed the gap.

Ergonomics and prompting

Prompting paradigm. Decoder-only models expect a single prompt string. You shove instructions, examples, context, and the task query into one sequence. This is flexible but fragile: prompt engineering becomes context engineering, and you pay for every token in the prompt at inference time.

Encoder-decoder prompting. You provide two distinct inputs: an encoder input (source text, document, context) and a decoder prompt (task prefix, target language token, or empty for pure generation). This separation is cleaner for pipeline stages — you can encode a document once, then run multiple decoder tasks (summarize, extract entities, translate) against the same encoder representation without re-encoding.

Structured output. Decoder-only models require constrained decoding (grammars, regex, JSON schemas) or heavy prompting to emit valid structure. Encoder-decoder models can be trained to emit structured formats directly (T5-style “translate English to German: …” prefixes), and the encoder-decoder boundary makes it easier to enforce constraints at the decoder level without polluting the encoder representation.

Ecosystem and tooling

Model availability. Decoder-only dominates open weight releases: LLaMA 3, Mistral, Qwen, Gemma, Phi, and their fine-tunes number in the thousands on Hugging Face. Encoder-decoder open weights cluster around T5, FLAN-T5, mT5, UL2, and BART variants — capable but fewer choices, especially at 7B+ parameters.

Inference engines. vLLM, TGI, TensorRT-LLM, and SGLang optimize heavily for decoder-only causal attention. Encoder-decoder support exists but lags: vLLM added encoder-decoder support in 2024, TensorRT-LLM supports T5-style models, but kernel optimizations (chunked prefill, prefix caching, speculative decoding) are decoder-first.

Fine-tuning tooling. LoRA/QLoRA recipes, PEFT configs, and trainer defaults assume decoder-only. Encoder-decoder fine-tuning works but requires more custom configuration — you need to decide whether to freeze the encoder, apply adapters to cross-attention, or tune both sides.

Tokenizers. Decoder-only models overwhelmingly use BPE/Unigram tokenizers trained on English-heavy corpora. Encoder-decoder models (especially mT5, mBART) often use SentencePiece with larger vocabularies covering 100+ languages, which helps multilingual tasks but increases embedding table size.

Hard limits and failure modes

Context window. Decoder-only models push context to 128K–1M+ tokens (LLaMA 3, Gemini, GPT-4). Encoder-decoder open weights typically cap at 4K–16K (T5: 512, mT5: 1024, FLAN-T5: 1024, UL2: 512, LongT5: 16K). The quadratic encoder attention makes long-context scaling harder. If you need 100K+ token context, decoder-only is your only practical choice among open models.

Hallucination profile. Decoder-only models hallucinate fluently — they generate plausible continuations unmoored from the prompt. Encoder-decoder models hallucinate differently: they may omit source details or conflate entities, but the encoder grounding reduces free-form fabrication. For retrieval-augmented generation where faithfulness matters, encoder-decoder models often score higher on factual consistency benchmarks.

Exposure bias. Both architectures suffer from exposure bias (training sees ground truth, inference sees model predictions), but decoder-only models compound it over longer generations. Encoder-decoder models reset at each decoder step with fresh cross-attention to the fixed encoder representation, which can stabilize long-form generation.

Beam search vs sampling. Encoder-decoder models were designed for beam search (translation, summarization). Decoder-only models work with beam search but it’s rarely used — sampling with temperature dominates. If your pipeline needs deterministic, high-quality beam outputs (e.g., for reranking or verification), encoder-decoder is more natural.

Which to choose

Choose decoder-only when:

• Building a general-purpose chat, coding, or reasoning agent
• You need 32K+ context window (long documents, multi-turn conversation, repo-scale code)
• You rely on in-context learning and few-shot prompting without fine-tuning
• You want maximum ecosystem support: inference engines, fine-tuning recipes, quantization tooling
• Your workload is open-ended generation where reasoning breadth matters more than bidirectional grounding

Choose encoder-decoder when:

• Building translation, summarization, or structured extraction pipelines
• Input is long but output is short (encode once, decode briefly)
• You need multilingual support with a single model (mT5, mBART cover 100+ languages natively)
• Faithfulness to source text is non-negotiable (RAG, legal, medical)
• You can fine-tune and want cleaner conditioning: encode the document once, run multiple decoder tasks
• You need deterministic beam search outputs for verification or reranking stages

Hybrid approach. Many production systems use both. A decoder-only model handles planning, reasoning, and open-ended generation. An encoder-decoder model (or a smaller decoder-only model with bidirectional encoding via prefix LM) handles faithful summarization, translation, and extraction over long contexts. Route by task, not religion.

The decoder-only vs encoder-decoder choice isn’t about which architecture is better — it’s about which failure modes you’d rather debug. Decoder-only fails by hallucinating fluently. Encoder-decoder fails by truncating context or requiring fine-tuning for every new task. Pick the failure mode your eval suite catches.