What is a transformer? The architecture behind every LLM

The transformer is a neural network architecture that processes sequences in parallel using self-attention rather than recurrence, enabling efficient training on massive datasets and forming the foundation of every modern large language model. Introduced in “Attention Is All You Need” (Vaswani et al., 2017), it replaced the sequential bottlenecks of RNNs and LSTMs with a mechanism that computes relationships between all tokens simultaneously. This parallelism, combined with the ability to model long-range dependencies without degradation, made large-scale language modeling tractable.

How the transformer works

At its core, the transformer stacks identical layers, each containing two sub-layers: a multi-head self-attention mechanism and a position-wise feed-forward network. Residual connections and layer normalization wrap each sub-layer.

Self-attention

Self-attention lets every token attend to every other token in the sequence. For an input matrix $X \in \mathbb{R}^{n \times d_{model}}$ (sequence length $n$, model dimension $d_{model}$), the layer projects $X$ into queries, keys, and values:

$$Q = XW^Q, \quad K = XW^K, \quad V = XW^V$$

Attention weights are computed as:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The scaling factor $\sqrt{d_k}$ prevents gradients from vanishing when $d_k$ is large. Multi-head attention runs this computation $h$ times in parallel with different learned projections, then concatenates the results:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$$

Each head learns different relational patterns — some track syntactic dependencies, others resolve coreference, others capture positional relationships.

Positional encoding

Since self-attention is permutation-invariant, the model needs explicit position information. The original paper uses fixed sinusoidal encodings:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

Modern implementations often use learned positional embeddings or rotary positional embeddings (RoPE), which encode position via rotation in the query/key space and generalize better to longer sequences.

Feed-forward network

Each position passes independently through a two-layer MLP with a non-linearity (typically GELU in modern variants, ReLU in the original):

$$\text{FFN}(x) = \text{GELU}(xW_1 + b_1)W_2 + b_2$$

The inner dimension is typically $4 \times d_{model}$. This provides the non-linear transformation capacity that attention alone cannot.

Encoder-decoder vs. decoder-only

The original transformer uses an encoder-decoder stack. The encoder processes the full input sequence bidirectionally. The decoder generates output autoregressively, using masked self-attention (preventing attention to future tokens) and cross-attention to the encoder output.

Modern LLMs (GPT, LLaMa, PaLM) use decoder-only architectures. They drop the encoder and cross-attention, training on next-token prediction over massive corpora. This simplifies the architecture and enables efficient inference with key-value caching.

Why the transformer matters

Parallelism over recurrence

RNNs process tokens sequentially: $h_t = f(h_{t-1}, x_t)$. This creates a hard dependency chain — step $t$ cannot start until step $t-1$ finishes. Transformers compute all token representations simultaneously. On GPU/TPU hardware, this translates to massive throughput gains during training.

Gradient flow

In deep RNNs, gradients propagate through many time steps, leading to vanishing or exploding gradients. The transformer’s residual connections provide direct gradient paths from output to any layer. Layer normalization stabilizes activation distributions. This allows training models with 100+ layers.

Long-range dependencies

Attention connects any two tokens in a single layer, regardless of distance. An RNN needs $O(n)$ steps to propagate information across $n$ tokens; a transformer does it in $O(1)$ layers. This is critical for tasks requiring global context — document summarization, code generation, long-form reasoning.

Scaling laws

Transformers exhibit predictable scaling: loss decreases as a power law with compute, parameters, and data. This predictability enabled the industry to invest confidently in training runs costing tens of millions of dollars. The architecture’s inductive biases (attention + MLP) are weak enough to not constrain learning at scale, yet strong enough to make optimization tractable.

Concrete example: a minimal transformer block

Here is a PyTorch implementation of a single decoder block, stripped to essentials:

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, n_heads: int, dropout: float = 0.1):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_k = d_model // n_heads

        self.w_q = nn.Linear(d_model, d_model, bias=False)
        self.w_k = nn.Linear(d_model, d_model, bias=False)
        self.w_v = nn.Linear(d_model, d_model, bias=False)
        self.w_o = nn.Linear(d_model, d_model, bias=False)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:
        batch, seq_len, _ = x.shape

        q = self.w_q(x).view(batch, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        k = self.w_k(x).view(batch, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        v = self.w_v(x).view(batch, seq_len, self.n_heads, self.d_k).transpose(1, 2)

        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))

        attn = F.softmax(scores, dim=-1)
        attn = self.dropout(attn)

        out = torch.matmul(attn, v)
        out = out.transpose(1, 2).contiguous().view(batch, seq_len, self.d_model)
        return self.w_o(out)


class FeedForward(nn.Module):
    def __init__(self, d_model: int, d_ff: int, dropout: float = 0.1):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)


class TransformerBlock(nn.Module):
    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.1):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, n_heads, dropout)
        self.ffn = FeedForward(d_model, d_ff, dropout)
        self.ln1 = nn.LayerNorm(d_model)
        self.ln2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:
        # Pre-norm architecture (more stable training)
        x = x + self.dropout(self.attention(self.ln1(x), mask))
        x = x + self.dropout(self.ffn(self.ln2(x)))
        return x

This block stacks into a full model by adding token embeddings, positional embeddings, a stack of TransformerBlock layers, a final layer norm, and an output projection to vocabulary size.

Common misconceptions

“Transformers use recurrence during inference”

False for the forward pass. During training, the entire sequence processes in parallel. During autoregressive generation, you do run the model sequentially — one token at a time — but each forward pass is still a single parallel computation over the current context. The key distinction: the architecture has no recurrent state carried between steps. Instead, the full context (via KV cache) is re-attended each step.

“Attention is O(n²) so transformers can’t handle long context”

The quadratic complexity is real for standard attention. However, the constant factors matter. FlashAttention (Dao et al., 2022) fuses the softmax and matrix multiplies into a single kernel, reducing memory traffic and making 8k–32k context practical on single GPUs. For longer context, architectures like sliding window attention, linear attention, or hybrid approaches (Mamba, RWKV) modify the attention pattern. The vanilla transformer remains the baseline; engineering optimizations extend its reach.

“The encoder-decoder structure is essential”

For translation and summarization, yes. For general-purpose LLMs, decoder-only won. Pre-training on next-token prediction with a causal mask learns representations that transfer to downstream tasks via prompting or fine-tuning. The encoder is unnecessary overhead. This simplification also enables techniques like prefix caching and speculative decoding that rely on the decoder’s autoregressive structure.

“Positional encodings are a solved problem”

Sinusoidal encodings generalize to unseen lengths but struggle with very long sequences. Learned embeddings cap maximum length. RoPE is now dominant but has failure modes at extreme lengths (the “lost in the middle” phenomenon). Relative positional biases (T5, ALiBi) offer alternatives. No single solution works universally; the choice depends on target context length and training budget.

“Bigger attention heads are always better”

Head dimension $d_k = d_{model} / n_{heads}$ trades off head capacity vs. number of heads. Too few heads limits the diversity of learned relations. Too many heads makes each head too low-dimensional to capture useful patterns. Empirically, 64–128 dimensions per head works well across scales. Some recent work (Grouped Query Attention, Multi-Query Attention) reduces key/value heads while keeping query heads, cutting KV cache memory with minimal quality loss.

The transformer in production

When you call an LLM API, you’re hitting a decoder-only transformer serving pipeline. Requests batch together; the model runs prefill (processing the prompt in parallel) then decode (generating tokens sequentially with KV cache). Techniques like continuous batching, paged attention (vLLM), and tensor parallelism keep GPUs utilized.

If you’re routing requests across multiple models or providers, the transformer’s standardized interface — token IDs in, logits out — is what makes interoperability possible. An OpenAI-compatible endpoint can serve LLaMa, Mistral, or proprietary models because they all share the same architectural contract. n4n.ai exposes this uniformity: one endpoint addressing 240+ models, with automatic fallback when a provider degrades, per-token usage metering, and passthrough of provider cache-control hints so your routing logic stays clean.