Encoder vs decoder transformers: what's the difference?

The encoder vs decoder transformer distinction isn’t academic — it determines what your model can actually do, how much compute you’ll burn, and which failure modes you’ll debug at 2 AM. Encoders build bidirectional representations for understanding tasks; decoders generate tokens left-to-right for synthesis tasks. Modern LLMs are decoder-only, but encoder-decoder and encoder-only architectures still dominate classification, retrieval, and structured prediction workloads.

Architecture and attention patterns

An encoder stack processes the entire input sequence simultaneously. Every position attends to every other position — full bidirectional self-attention. This means token i sees token j and token j sees token i in the same layer. BERT, RoBERTa, and DeBERTa exploit this for tasks where context from both directions matters: named entity recognition, sentiment classification, semantic similarity.

A decoder stack masks future positions. Token i attends only to tokens ≤ i. This causal masking enables autoregressive generation: you feed the model a prefix, it predicts the next token, you append it, repeat. GPT, LLaMA, and their descendants are decoder-only. Encoder-decoder models (T5, BART, FLAN-T5) combine both: an encoder ingests the source, a decoder generates the target conditioned on the encoder’s output via cross-attention.

# Encoder self-attention: no mask, all positions visible
def encoder_self_attention(q, k, v):
    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)
    return softmax(scores) @ v

# Decoder self-attention: causal mask prevents peeking forward
def decoder_self_attention(q, k, v):
    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)
    mask = torch.triu(torch.ones_like(scores), diagonal=1).bool()
    scores.masked_fill_(mask, -float('inf'))
    return softmax(scores) @ v

The practical consequence: encoders produce one fixed-size representation per input (or per token if you keep the sequence). Decoders produce a distribution over the vocabulary at each step, conditioned on everything generated so far. You cannot trivially swap one for the other.

Training objectives and what they optimize for

Encoders train with masked language modeling (MLM) or denoising objectives. Random tokens are corrupted; the model reconstructs them using bidirectional context. This learns rich contextual embeddings but not coherent generation. The loss function doesn’t penalize incoherent continuations — it only cares about predicting the masked positions correctly.

Decoders train with next-token prediction (causal language modeling). The objective is exactly the generation task: minimize cross-entropy of the true next token given the prefix. This aligns training and inference but creates exposure bias — the model never sees its own mistakes during training, so errors compound at generation time.

Encoder-decoder models typically use span corruption or prefix language modeling. T5 masks random spans in the input and asks the decoder to reconstruct them. This teaches the encoder to compress information the decoder will need, and the decoder to condition on that compressed representation.

# MLM loss (encoder): only compute loss on masked positions
def mlm_loss(logits, labels, mask_positions):
    masked_logits = logits[mask_positions]
    masked_labels = labels[mask_positions]
    return cross_entropy(masked_logits, masked_labels)

# Causal LM loss (decoder): compute loss on all positions
def clm_loss(logits, labels):
    shift_logits = logits[..., :-1, :].contiguous()
    shift_labels = labels[..., 1:].contiguous()
    return cross_entropy(shift_logits.view(-1, vocab_size), shift_labels.view(-1))

Inference characteristics: latency, throughput, memory

Encoder inference is a single forward pass. Latency scales with sequence length quadratically in attention (though flash attention and linear attention variants mitigate this). Throughput is predictable: one pass, done. You can batch aggressively because every request has the same compute graph. Memory scales with batch size × sequence length × hidden dimension.

Decoder inference is iterative. Each token requires a full forward pass through the model. Latency scales linearly with generated tokens but quadratically with total context length (prompt + generated so far). KV caching avoids recomputing attention over the prompt, but the cache grows with each generated token — memory scales with batch size × (prompt length + generated length) × layers × heads × head dimension.

This creates a fundamental throughput asymmetry. An encoder classifies 1,000 documents in roughly the same time it takes a decoder to generate 1,000 tokens total across all requests. For high-volume classification or embedding workloads, encoders are dramatically cheaper.

# Encoder: single forward pass
embeddings = encoder(input_ids, attention_mask)  # [batch, seq, hidden]
logits = classifier(embeddings[:, 0])  # CLS token
# Done. One kernel launch per layer.

# Decoder: iterative with KV cache
past_key_values = None
generated = []
for _ in range(max_new_tokens):
    logits, past_key_values = decoder(input_ids, past_key_values=past_key_values)
    next_token = sample(logits[:, -1])
    generated.append(next_token)
    input_ids = next_token.unsqueeze(1)  # only feed new token next step
# max_new_tokens kernel launches per layer.

Capabilities and task fit

Encoders win at: text classification, sentiment analysis, NER, semantic search, duplicate detection, entailment, any task where the output is a label or fixed-size vector.

Decoders win at: open-ended generation, code completion, chat, reasoning chains, tool use, any task where the output length is unbounded or the format is free-form.

Encoder-decoder wins at: translation, summarization, question answering with long contexts, any conditional generation where the input is long and the output is structured but variable-length.

Ecosystem and tooling

The decoder-only ecosystem has absorbed most engineering investment since 2022. Hugging Face transformers, vLLM, TGI, TensorRT-LLM, and llama.cpp all optimize for causal generation first. Quantization (AWQ, GPTQ, GGUF), speculative decoding, continuous batching, and prefix caching target decoder workloads.

Encoder tooling is mature but quieter. sentence-transformers wraps BERT-family models for embeddings with pooling strategies, similarity search integration (FAISS, Annoy, Milvus), and ONNX export. Fine-tuning libraries (SetFit, simple transformers) assume classification heads. You won’t find speculative decoding for encoders because there’s no generation loop to accelerate.

Encoder-decoder sits in an awkward middle. T5 and FLAN-T5 checkpoints are widely available, but inference engines treat them as second-class citizens. vLLM supports encoder-decoder but with less optimization than decoder-only. Most production teams either fine-tune a decoder with prompt templates or use an encoder for understanding + a separate decoder for generation.

Cost model in practice

If you’re calling a hosted API, the pricing model reflects the compute reality. Embedding endpoints (encoder) charge per 1K input tokens — often $0.0001–0.001. Generation endpoints (decoder) charge per 1K input plus per 1K output tokens — typically 10–100× more expensive. A 10K token document classified by an encoder costs ~$0.001. The same document summarized by a decoder costs ~$0.05–0.30.

Self-hosted changes the calculus but not the ratio. An encoder on a T4 classifies ~500 sequences/second at 512 tokens. A 7B decoder on the same GPU generates ~50 tokens/second. If your workload is “embed 1M documents nightly,” the encoder finishes in ~33 minutes. The decoder doing “summarize 1M documents” takes ~5.5 hours on the same hardware.

Limits and failure modes

Encoders hit a hard ceiling at their pretrained max sequence length (usually 512 for BERT, 4K–32K for Longformer/BigBird variants). Extending requires position interpolation or architectural changes (ALiBi, RoPE). They also struggle with tasks requiring output structure — you must design a label schema or add a decoder head.

Decoders hallucinate, drift, and repeat. They’re sensitive to prompt formatting. Long-context performance degrades even with 128K+ windows — the “lost in the middle” phenomenon is real. KV cache memory limits practical batch sizes at long contexts. Quantization below 4-bit often breaks reasoning.

Encoder-decoder inherits both: encoder length limits on the input, decoder generation failures on the output. Cross-attention adds quadratic cost in (source length × target length), making very long inputs with very long outputs expensive.

Which to choose

Choose encoder-only when:

• Your output is a label, score, or fixed-size vector
• You need embeddings for retrieval, clustering, or similarity
• Throughput and cost per document matter more than flexibility
• You have labeled data for fine-tuning (even a few hundred examples)
• Tasks: classification, NER, entailment, semantic search, deduplication, reranking

Choose decoder-only when:

• Your output is free-form text of variable length
• You need reasoning, code generation, or tool use
• Few-shot prompting beats fine-tuning for your iteration speed
• You can absorb higher per-request latency and cost
• Tasks: chat, summarization (via prompting), code, agents, creative writing, data extraction with flexible schemas

Choose encoder-decoder when:

• You have a clear source→target mapping with long sources
• Output structure is predictable but length varies
• You need to condition generation on a long document without stuffing it into a decoder prompt
• Tasks: translation, document-grounded QA, controlled summarization, data-to-text

Hybrid pattern (common in production): Use an encoder for understanding (classification, routing, retrieval, entity extraction) and a decoder for synthesis (response generation, explanation, formatting). This separates the latency-critical path from the quality-critical path. The encoder runs at ~10ms/request; the decoder runs at ~500ms/request. You scale them independently.

The encoder vs decoder transformer decision is ultimately about the shape of your data and the economics of your workload. If you’re building a classifier, don’t reach for a 7B decoder because it’s “what everyone uses.” If you’re building a chatbot, don’t force an encoder-decoder because you read T5 was efficient. Match the architecture to the task, then optimize the inference stack for that architecture.