The history of the transformer architecture reads like a series of bets that paid off: attention over recurrence, scale over inductive bias, pretraining over task-specific engineering. Each bet shifted what was computationally tractable and economically viable. Understanding this lineage isn’t academic — it explains why your inference latency looks the way it does, why context windows grew the way they did, and where the next bottlenecks will appear.
The 2017 baseline: attention as a replacement for recurrence
The original Transformer paper introduced a deceptively simple idea: replace the sequential dependency of RNNs and LSTMs with parallelizable self-attention. The key insight wasn’t attention itself — Bahdanau and Luong had already popularized it for machine translation — but making attention the entire architecture.
# Simplified scaled dot-product attention
def attention(Q, K, V, mask=None):
# Q: (batch, seq_len, d_model)
# K, V: (batch, seq_len, d_model)
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attn = F.softmax(scores, dim=-1)
return torch.matmul(attn, V)This changed the compute profile fundamentally. RNNs forced O(seq_len) sequential steps; transformers reduced that to O(1) depth with O(seq_len²) memory. For training, this was a win — GPUs love parallel matmuls. For inference, it introduced the KV cache problem we still fight today: each generated token requires attending to all previous tokens, making generation O(seq_len) in memory and compute per step.
The 2017 model used 6 encoder and 6 decoder layers, 512-dimensional embeddings, 8 attention heads. It fit on 8 GPUs for 3.5 days. That scale feels quaint now, but the architectural decisions — residual connections, layer norm, positional encodings — survived virtually unchanged.
BERT and the encoder-only detour (2018)
BERT proved you could pretrain a transformer encoder on masked language modeling and fine-tune it for everything. The architecture was nearly identical to the original encoder stack, but the training objective changed the economics: one expensive pretrain, many cheap fine-tunes.
# BERT pretraining objective (simplified)
def bert_loss(logits, labels, mask):
# logits: (batch, seq_len, vocab_size)
# labels: (batch, seq_len)
# mask: (batch, seq_len) -- 1 for masked positions
loss_fct = nn.CrossEntropyLoss(ignore_index=-100)
active_loss = mask.view(-1) == 1
active_logits = logits.view(-1, vocab_size)[active_loss]
active_labels = labels.view(-1)[active_loss]
return loss_fct(active_logits, active_labels)BERT-base (110M params) and BERT-large (340M) established that bidirectional context mattered more than autoregressive generation for understanding tasks. But the encoder-only design hit a ceiling: you couldn’t generate coherent long-form text. The field needed decoder-only models.
GPT-1 and GPT-2: the decoder-only bet (2018-2019)
Radford et al. showed that a decoder-only transformer trained on next-token prediction could do zero-shot task transfer. GPT-1 (117M params) was a proof of concept. GPT-2 (1.5B params) demonstrated that scaling the same architecture — more layers, wider embeddings, more data — yielded qualitatively new capabilities without architectural changes.
The GPT-2 release strategy (staged release due to “safety concerns”) was controversial, but the technical lesson was clear: the decoder-only architecture with causal masking was sufficient for both understanding and generation.
# Causal mask for decoder-only attention
def causal_mask(seq_len, device):
mask = torch.triu(torch.ones(seq_len, seq_len, device=device), diagonal=1)
return mask == 0 # True = attend, False = masked
# In practice, fused into attention kernels
# FlashAttention computes this implicitly without materializing the maskGPT-2 also introduced the tokenizer that became the de facto standard: byte-pair encoding (BPE) with a 50,257 token vocabulary. That tokenizer choice still constrains every GPT-model descendant — including how they handle code, non-English languages, and arithmetic.
GPT-3: scaling laws and in-context learning (2020)
GPT-3 (175B params) was the first model where the training run itself became the story. The Kaplan et al. scaling laws paper (published alongside GPT-3) formalized what practitioners had observed: test loss follows a power law in compute, dataset size, and model size.
L(N, D, C) ≈ (N_c/N)^α_N + (D_c/D)^α_D + L_∞This meant you could predict the loss of a 175B model from 1B and 10B runs. The industry started treating model training as a capital expenditure problem with predictable returns.
GPT-3’s emergent capability was in-context learning: the model could perform tasks from a few examples in the prompt, no gradient updates required. This reframed the transformer from a “model you fine-tune” to a “model you prompt.” The API economy followed.
But GPT-3 exposed hard limits:
- 2048 token context window (a hard constraint from O(seq_len²) attention)
- No instruction following — it completed patterns, not requests
- Inference cost: 175B params required 8×A100 80GB just to load weights in FP16
The architecture refinements: 2021-2022
Between GPT-3 and GPT-4, several architectural changes became standard. None were as visible as “attention is all you need,” but each mattered for training stability and inference efficiency.
Pre-norm vs post-norm
The original transformer used post-norm (residual → layer norm). Pre-norm (layer norm → residual) stabilizes training at depth by preventing gradient explosion in the residual stream.
# Post-norm (original)
x = layer_norm(x + sublayer(x))
# Pre-norm (now standard)
x = x + sublayer(layer_norm(x))Pre-norm lets you train 100+ layer models without learning rate warmup tricks. Every major model since ~2021 uses it.
Rotary positional embeddings (RoPE)
Absolute positional encodings (sinusoidal or learned) don’t generalize beyond training length. RoPE encodes position by rotating query and key vectors in complex space, making attention scores depend on relative position naturally.
# RoPE: rotate Q and K by position-dependent angles
def apply_rope(q, k, pos, dim):
# q, k: (batch, seq_len, n_heads, head_dim)
# pos: (seq_len,)
freqs = 1.0 / (10000 ** (torch.arange(0, dim, 2) / dim))
angles = pos[:, None] * freqs[None, :] # (seq_len, dim/2)
sin, cos = angles.sin(), angles.cos()
# Interleave sin/cos for complex rotation
q_rot = rotate_half(q) * sin + q * cos
k_rot = rotate_half(k) * sin + k * cos
return q_rot, k_rotRoPE enables extrapolation to longer contexts at inference time — a prerequisite for the 32K and 128K windows we see now.
SwiGLU and grouped-query attention
SwiGLU (Swish-Gated Linear Unit) replaced the standard FFN (GeLU + linear) with a gated variant that improves training dynamics:
# Standard FFN
x = gelu(linear1(x))
x = linear2(x)
# SwiGLU
x = silu(linear1(x)) * linear2(x) # gate * value
x = linear3(x)Grouped-query attention (GQA) reduced the KV cache memory by sharing keys and values across multiple query heads. With 32 query heads and 8 KV heads, you get 4× KV cache savings with minimal quality loss — critical for long-context inference.
# GQA: repeat KV heads to match query heads
def repeat_kv(kv, n_rep):
# kv: (batch, seq_len, n_kv_heads, head_dim)
batch, seq_len, n_kv_heads, head_dim = kv.shape
kv = kv[:, :, :, None, :].expand(batch, seq_len, n_kv_heads, n_rep, head_dim)
return kv.reshape(batch, seq_len, n_kv_heads * n_rep, head_dim)These changes compounded. A 2022-era 7B model with pre-norm, RoPE, SwiGLU, and GQA outperformed a 2020-era 13B model on the same compute budget.
Instruction tuning and RLHF: aligning the base model
The base model predicts next tokens on internet text. That’s not what users want. Instruction tuning (supervised fine-tuning on prompt-response pairs) and RLHF (reinforcement learning from human feedback) closed the gap.
# Instruction tuning format (Alpaca style)
{
"instruction": "Write a Python function that...",
"input": "",
"output": "def fibonacci(n):\n if n <= 1:\n return n\n return fibonacci(n-1) + fibonacci(n-2)"
}
# RLHF: PPO on a reward model
# 1. Train reward model r_θ(prompt, response) on human comparisons
# 2. PPO: maximize E[r_θ(prompt, π_φ(prompt))] - β * KL(π_φ || π_ref)The key insight: a relatively small amount of high-quality instruction data (50K-100K examples) transforms a base model into a usable assistant. The base model’s capabilities are largely preserved; the alignment layer just steers them.
GPT-4: the mixture-of-experts leap (2023)
GPT-4’s architecture was never officially disclosed, but credible reporting and reverse-engineering converge on a sparse mixture-of-experts (MoE) model: roughly 1.8T total parameters across 16 experts, with 2 experts active per token (~360B active params).
# Simplified MoE layer
class MoELayer(nn.Module):
def __init__(self, n_experts, top_k, d_model, d_ff):
super().__init__()
self.gate = nn.Linear(d_model, n_experts, bias=False)
self.experts = nn.ModuleList([
nn.Sequential(
nn.Linear(d_model, d_ff),
nn.GELU(),
nn.Linear(d_ff, d_model)
) for _ in range(n_experts)
])
self.top_k = top_k
def forward(self, x):
# x: (batch, seq_len, d_model)
gate_logits = self.gate(x) # (batch, seq_len, n_experts)
weights, indices = torch.topk(gate_logits, self.top_k, dim=-1)
weights = F.softmax(weights, dim=-1)
# Dispatch to experts (simplified; real impl uses scatter/gather)
out = torch.zeros_like(x)
for i in range(self.top_k):
expert_idx = indices[..., i]
expert_weight = weights[..., i:i+1]
# In practice: batched expert computation with capacity factors
expert_out = self.experts[expert_idx](x)
out += expert_weight * expert_out
return outMoE changes the scaling economics: you get the representational capacity of a 1.8T dense model at the compute cost of a 360B model (for training FLOPs) and the memory cost of a 360B model (for inference, since only active experts’ weights are loaded). The tradeoff: expert routing adds communication overhead, and training stability requires careful load balancing.
# Load balancing loss (auxiliary loss)
def load_balancing_loss(gate_logits, top_k):
# Encourage uniform expert utilization
probs = F.softmax(gate_logits, dim=-1) # (batch, seq_len, n_experts)
top_probs, top_indices = torch.topk(probs, top_k, dim=-1)
# Fraction of tokens routed to each expert
expert_mask = F.one_hot(top_indices, num_classes=n_experts).float()
tokens_per_expert = expert_mask.mean(dim=(0, 1)) # (n_experts,)
# Target: uniform 1/n_experts
loss = n_experts * (tokens_per_expert * probs.mean(dim=(0, 1))).sum()
return lossGPT-4 also expanded context to 32K (8K at launch) and added native multimodal support — image inputs processed by a vision encoder whose outputs are projected into the language model’s embedding space. The vision encoder is likely a ViT variant trained contrastively (CLIP-style), then aligned via adapter layers.
What this history means for inference today
The architectural lineage dictates your production constraints:
KV cache is the bottleneck. Every decoder-only transformer since GPT-1 stores K and V for all previous tokens. At 32K context with 32 layers, 32 heads, 128 head_dim, FP16: 32 × 32 × 128 × 2 × 32K × 2 bytes ≈ 4.2 GB per request. This is why batched inference and paged attention (vLLM) matter — they’re direct responses to the O(seq_len) memory growth baked into the 2017 architecture.
Tokenization is frozen. The GPT-2 BPE tokenizer persists because retraining it would invalidate every downstream artifact: embeddings, positional encodings, vocabulary projections. You work around its quirks (poor compression for code, arithmetic tokenization) rather than fix them.
Context window expansion requires kernel changes. FlashAttention (2022) and FlashAttention-2 (2023) made 32K+ contexts tractable by fusing the attention computation and avoiding the O(seq_len²) materialization of the attention matrix. Without kernel-level work, the quadratic memory wall wins.
MoE shifts the serving profile. Dense models have uniform per-token latency. MoE models have variable latency depending on expert routing and batch scheduling. Load balancing across experts becomes a serving infrastructure problem, not just a training one.
The decisive takeaway
The transformer architecture hasn’t fundamentally changed since 2017. What changed is the willingness to scale it, the engineering to make scaling efficient, and the post-training recipes to make the raw capability usable. Every “new architecture” paper since then has either failed to displace the transformer or been absorbed as an incremental improvement (RoPE, SwiGLU, GQA, MoE).
If you’re building on LLMs today, bet on the decoder-only transformer with causal attention, RoPE, SwiGLU, and GQA — possibly MoE at scale. The next breakthrough won’t be a new attention mechanism. It will be a training or inference innovation that changes the compute economics, just as FlashAttention and MoE did. The architecture is settled; the engineering is where the leverage lives.