How GPT's transformer architecture differs from BERT's

GPT and BERT both descend from the 2017 “Attention Is All You Need” transformer, but they diverge at the architectural level in ways that dictate everything from training cost to inference latency. The GPT vs BERT architecture distinction comes down to a single structural choice: GPT stacks decoder blocks with causal masking, while BERT stacks encoder blocks with bidirectional attention. That difference cascades into how you prompt them, how you fine-tune them, and which production workloads they serve.

Architecture: decoder-only versus encoder-only

GPT uses a decoder-only stack. Each layer applies masked self-attention where position i can only attend to positions ≤ i, followed by a feed-forward network. The mask is baked into the attention scores before softmax:

def causal_mask(seq_len, device):
    # Upper-triangular matrix with -inf above diagonal
    mask = torch.triu(torch.ones(seq_len, seq_len, device=device), diagonal=1)
    return mask.masked_fill(mask == 1, float('-inf'))

# Inside attention
attn_scores = (q @ k.transpose(-2, -1)) / math.sqrt(head_dim)
attn_scores = attn_scores + causal_mask(seq_len, q.device)
attn_weights = F.softmax(attn_scores, dim=-1)

BERT uses an encoder-only stack. Every position attends to every other position in the same layer — no masking. The attention pattern is fully bidirectional from layer one.

# BERT attention — no causal mask
attn_scores = (q @ k.transpose(-2, -1)) / math.sqrt(head_dim)
attn_weights = F.softmax(attn_scores, dim=-1)  # all positions visible

This structural difference means GPT models generate tokens left-to-right by design. BERT models cannot generate sequentially without architectural surgery; they produce a single contextualized representation per input token.

Training objectives: next-token prediction versus masked language modeling

GPT minimizes cross-entropy on next-token prediction across a massive corpus. The loss at position t depends only on tokens < t:

# GPT loss — standard language modeling
logits = model(input_ids)  # [batch, seq_len, vocab]
shift_logits = logits[:, :-1, :].contiguous()
shift_labels = input_ids[:, 1:].contiguous()
loss = F.cross_entropy(shift_logits.view(-1, vocab_size), shift_labels.view(-1))

BERT minimizes two objectives jointly: masked language modeling (MLM) and next-sentence prediction (NSP). For MLM, 15% of tokens are replaced — 80% with [MASK], 10% with random tokens, 10% unchanged — and the model predicts the original token IDs at masked positions using full bidirectional context.

# BERT MLM loss — only compute loss on masked positions
logits = model(input_ids, attention_mask=attn_mask)  # [batch, seq_len, vocab]
mlm_loss = F.cross_entropy(
    logits.view(-1, vocab_size),
    labels.view(-1),
    ignore_index=-100  # -100 on non-masked positions
)

NSP was dropped in later variants (RoBERTa, DeBERTa) after ablation studies showed it added little. Modern BERT-style models train on MLM alone, often with dynamic masking per epoch.

Inference patterns and latency profiles

GPT inference is autoregressive. Generating N tokens requires N forward passes, each growing the KV cache by one position. Latency scales linearly with output length. Throughput is bounded by memory bandwidth for the KV cache, not compute.

# GPT generation loop (simplified)
kv_cache = None
for _ in range(max_new_tokens):
    logits, kv_cache = model(next_token_id, past_key_values=kv_cache, use_cache=True)
    next_token_id = logits[:, -1, :].argmax(dim=-1)
    generated.append(next_token_id)

BERT inference is a single forward pass. You feed the full sequence (up to 512 tokens for base BERT) and get contextualized embeddings for every position simultaneously. Latency is constant regardless of “output length” because there is no generation — you extract representations and attach a task head.

# BERT feature extraction — one forward pass
with torch.no_grad():
    outputs = model(input_ids, attention_mask=attn_mask)
    # last_hidden_state: [batch, seq_len, hidden_dim]
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [CLS] token
    token_embeddings = outputs.last_hidden_state[:, 1:-1, :]  # exclude [CLS], [SEP]

For classification or extraction tasks, BERT-style models are 10-50x faster at inference than GPT-style models of comparable parameter count. For generation tasks, GPT is the only viable option without architectural modification.

Fine-tuning strategies

GPT fine-tuning typically uses full-parameter supervised fine-tuning (SFT) on instruction-response pairs, followed by preference optimization (DPO, PPO). LoRA adapters are common for parameter-efficient tuning:

# LoRA on GPT — inject low-rank adapters into attention projections
class LoRALinear(nn.Module):
    def __init__(self, base_layer, rank=16, alpha=32):
        super().__init__()
        self.base = base_layer
        self.rank = rank
        self.scale = alpha / rank
        self.lora_A = nn.Linear(base_layer.in_features, rank, bias=False)
        self.lora_B = nn.Linear(rank, base_layer.out_features, bias=False)
        nn.init.kaiming_uniform_(self.lora_A.weight, a=math.sqrt(5))
        nn.init.zeros_(self.lora_B.weight)

    def forward(self, x):
        return self.base(x) + self.scale * self.lora_B(self.lora_A(x))

BERT fine-tuning replaces the [CLS] projection head (or token-level heads for NER/QA) and trains on labeled data. Full fine-tuning is standard because the parameter count is smaller (110M-340M for base/large) and the encoder converges quickly. LoRA works but is less common — the compute savings matter less when a full fine-tune finishes in hours on a single GPU.

# BERT classification head — typical fine-tune setup
class BertForSequenceClassification(nn.Module):
    def __init__(self, bert_model, num_labels, dropout=0.1):
        super().__init__()
        self.bert = bert_model
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(bert_model.config.hidden_size, num_labels)

    def forward(self, input_ids, attention_mask=None, labels=None):
        outputs = self.bert(input_ids, attention_mask=attention_mask)
        pooled = outputs.last_hidden_state[:, 0]  # [CLS]
        logits = self.classifier(self.dropout(pooled))
        loss = None
        if labels is not None:
            loss = F.cross_entropy(logits, labels)
        return {"loss": loss, "logits": logits}

Context windows and positional encoding

Original GPT-2/3 used learned absolute positional embeddings with a 1024/2048 token limit. Modern GPT variants (GPT-NeoX, LLaMA, GPT-4) use RoPE (Rotary Positional Embeddings), which extends to 4K-128K+ tokens by rotating query/key vectors in complex space:

# RoPE — rotate q, k by position-dependent frequencies
def apply_rope(q, k, pos, dim, base=10000):
    # q, k: [batch, heads, seq_len, head_dim]
    # pos: [seq_len]
    freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=q.device).float() / dim))
    angles = pos[:, None] * freq[None, :]  # [seq_len, dim/2]
    sin, cos = angles.sin(), angles.cos()
    # Interleave sin/cos for complex rotation
    q_rot = torch.stack([-q[..., 1::2], q[..., ::2]], dim=-1).reshape_as(q)
    k_rot = torch.stack([-k[..., 1::2], k[..., ::2]], dim=-1).reshape_as(k)
    q = q * cos + q_rot * sin
    k = k * cos + k_rot * sin
    return q, k

BERT used learned absolute positions capped at 512 tokens. Long-context BERT variants (Longformer, BigBird, DeBERTa-v3) introduce sparse attention patterns or relative positional biases to scale to 4K-16K tokens, but the 512-token ceiling remains the default for most off-the-shelf checkpoints.

Model sizes and compute requirements

The parameter counts reflect historical conventions. Nothing prevents training a 7B-parameter encoder-only model or a 110M-parameter decoder-only model — but the ecosystem tooling, checkpoints, and literature cluster around these ranges.

Ecosystem and tooling

GPT-style models dominate the open LLM landscape: LLaMA, Mistral, Qwen, Gemma, Phi, Falcon. Hugging Face transformers provides AutoModelForCausalLM, generate(), and PEFT/LoRA integrations. vLLM, TGI, and TensorRT-LLM optimize autoregressive serving with continuous batching, PagedAttention, and speculative decoding.

BERT-style models dominate classification, extraction, and retrieval: bert-base-uncased, roberta-large, deberta-v3-large, e5-base, bge-large. Tooling centers on AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoModelForQuestionAnswering, and sentence-transformers for embeddings. ONNX Runtime and TensorRT optimize encoder inference aggressively — batch sizes of 512+ are routine on a single GPU.

If you need embeddings for RAG, use a BERT-style bi-encoder (E5, BGE, GTE). If you need a reranker, use a BERT-style cross-encoder. If you need generation, summarization, or open-ended reasoning, use GPT-style.

Which to choose

Choose GPT-style (decoder-only) when:

• You need free-form text generation: chat, code completion, creative writing, agent loops.
• The task requires reasoning over long contexts (16K+ tokens) with RoPE-based models.
• You need few-shot or in-context learning without gradient updates.
• You’re building a general-purpose assistant or copilot.

Choose BERT-style (encoder-only) when:

• You need classification, NER, sentiment, or intent detection — attach a head and fine-tune on labeled data.
• You need token-level labels: span extraction, POS tagging, QA span prediction.
• You need dense embeddings for retrieval, clustering, or semantic search — use a bi-encoder.
• You need a cross-encoder reranker for precision at top-k.
• Latency and throughput are critical: classification at 10K+ req/s on a single A100 is routine.
• Your labeled dataset is small (hundreds to thousands of examples) — encoder fine-tuning converges fast.

Hybrid architectures exist but serve narrow niches:

• Encoder-decoder (T5, BART, FLAN-T5): seq2seq tasks where output length is unpredictable but bounded (translation, summarization). Fine-tunes like BERT, generates like GPT.
• Prefix-LM (U-PaLM, GLM): bidirectional prefix + causal suffix. Useful for infilling but rarely deployed standalone.
• Decoder with bidirectional attention (Mamba, RWKV): not transformers, but worth knowing if you’re evaluating architecture alternatives.

The GPT vs BERT architecture decision is rarely about which is “better” — it’s about matching the inductive bias to the task. Decoder-only models internalize the generative distribution p(xₜ | xₜ₋₁…x₁). Encoder-only models internalize contextual representations hᵢ = f(x₁…xₙ). If your loss function aligns with the former, use GPT. If it aligns with the latter, use BERT.