What are LLM parameters, exactly?

LLM parameters are the learned numerical weights and biases inside a neural network that determine how the model transforms input tokens into output probabilities. In transformer architectures, these parameters live primarily in attention projection matrices, feed-forward networks, and layer normalization layers — totaling billions of floating-point values that get updated during training. The parameter count directly constrains the model’s capacity to represent patterns, but it is not a standalone proxy for quality.

How parameters work in a transformer

Every transformer block contains a fixed set of parameterized operations. The multi-head attention module projects the residual stream into query, key, and value vectors using three learned weight matrices (W_q, W_k, W_v), then projects the concatenated attention output through W_o. The position-wise feed-forward network applies two linear transformations with a non-linearity between them — typically GeLU or SwiGLU. Layer normalization adds scale and shift parameters per feature dimension. Embedding tables (token and positional) and the final unembedding (lm_head) account for a significant fraction of total parameters, especially at large vocabularies.

# Simplified parameter accounting for a single transformer block
# d_model = hidden size, n_heads = attention heads, d_ff = feed-forward width
# vocab_size = tokenizer vocabulary size

def count_block_params(d_model, n_heads, d_ff, vocab_size):
    # Attention projections: Q, K, V, O
    attn_params = 4 * d_model * d_model
    
    # Feed-forward: up-projection, down-projection (SwiGLU uses 3 matrices)
    ffn_params = 3 * d_model * d_ff
    
    # LayerNorm: scale + bias per feature (2 per LN, 2 LNs per block)
    ln_params = 4 * d_model
    
    return attn_params + ffn_params + ln_params

# Embedding + output head (often tied)
embedding_params = 2 * vocab_size * d_model  # tied weights = 1x

For a 7B parameter model like Llama 2 7B: d_model=4096, n_heads=32, d_ff=11008, vocab_size=32000, n_layers=32. The embedding tables alone contribute ~260M parameters. The remaining ~6.7B distribute across 32 layers. This arithmetic explains why widening the feed-forward dimension (d_ff) is more parameter-efficient than adding layers — you get more compute per parameter.

Why parameter count matters

Parameter count correlates with two distinct properties: memorization capacity and in-context learning ability. Larger models can store more training data patterns in their weights, reducing the need to rely on retrieval at inference time. They also develop stronger induction heads — attention patterns that copy and complete sequences — which enables few-shot learning without weight updates.

The scaling laws established by Kaplan et al. (2020) and refined by Hoffmann et al. (2022, Chinchilla) show that optimal training compute allocation follows a power law: for a given compute budget C, the optimal parameter count N scales as C^0.5 and optimal tokens D scales as C^0.5. Undertraining a large model (too few tokens per parameter) wastes capacity; overtraining a small model hits diminishing returns.

{
  "chinchilla_optimal": {
    "compute_flops": 1e21,
    "optimal_params": 6.7e9,
    "optimal_tokens": 1.4e12,
    "params_per_token": 4.8
  },
  "implication": "A 7B model needs ~1.4T tokens for compute-optimal training. Most open models are undertrained relative to this curve."
}

But parameter count alone doesn’t determine utility. Architecture choices (grouped-query attention, SwiGLU, RoPE), data quality, training stability, and post-training (SFT, RLHF, DPO) shift the capability curve significantly. A well-trained 7B model can outperform a poorly trained 13B model on downstream tasks.

Concrete example: parameter breakdown of a real model

Llama 3 8B (actually 8.03B parameters) illustrates the distribution:

The attention matrices dominate because each head projects the full d_model dimension. Grouped-query attention (GQA) reduces this by sharing K/V projections across heads — Llama 3 8B uses 8 key/value heads for 32 query heads, cutting attention KV parameters by 4x with minimal quality loss.

# GQA parameter savings calculation
def gqa_savings(d_model, n_q_heads, n_kv_heads):
    # Standard MHA: 3 * d_model * d_model (Q, K, V) + d_model * d_model (O)
    mha = 4 * d_model * d_model
    
    # GQA: Q uses all heads, K/V share across groups
    head_dim = d_model // n_q_heads
    gqa = (n_q_heads * head_dim * d_model) + \
          (2 * n_kv_heads * head_dim * d_model) + \
          (d_model * d_model)  # O projection unchanged
    return mha - gqa

# Llama 3 8B: d_model=4096, n_q_heads=32, n_kv_heads=8
# Savings: ~1.0B parameters in attention alone

At inference time, parameter count determines memory footprint and compute per token. An 8B model at BF16 (2 bytes/param) needs ~16 GB VRAM for weights alone, plus KV cache. At 4-bit quantization (GPTQ/AWQ), that drops to ~4.5 GB — runnable on a 24 GB consumer GPU with headroom for context. The same model at FP16 would need ~32 GB for weights + KV cache at 4K context.

Common misconceptions

Misconception: “More parameters = smarter model.”
Parameter count is necessary but not sufficient. A 70B model trained on low-quality data with unstable optimization will underperform a 7B model trained on curated data with proper learning rate schedules. The Phi and Gemma families demonstrate that sub-10B models can match or exceed older 70B models on benchmarks when data and training are optimized.

Misconception: “Parameter count equals model size on disk.”
Quantization, weight tying, and serialization format change disk size independently of parameter count. An 8B parameter model:

• FP32: ~32 GB
• BF16/FP16: ~16 GB
• 4-bit quantized: ~4.5 GB
• 3-bit quantized: ~3.5 GB
  All have identical parameter counts. The safetensors format adds minimal overhead; PyTorch .bin checkpoints add more.

Misconception: “All parameters are active for every forward pass.”
Mixture-of-experts (MoE) models like Mixtral 8x7B have 47B total parameters but only 13B active per token (2 of 8 experts routed per layer). The parameter count in the model card refers to total parameters, not active parameters. This distinction matters for both memory planning and FLOPs estimation.

# MoE active parameter calculation
def moe_active_params(total_params, n_experts, experts_per_token):
    # Non-expert params (embeddings, attention, norms, router) are always active
    # Expert params (FFN) are only active for selected experts
    expert_fraction = experts_per_token / n_experts
    # Roughly: ~60% of params in FFN experts for typical MoE
    ffn_expert_ratio = 0.6
    always_active = total_params * (1 - ffn_expert_ratio)
    conditionally_active = total_params * ffn_expert_ratio * expert_fraction
    return always_active + conditionally_active

# Mixtral 8x7B: 47B total, 8 experts, top-2 routing
# Active: ~13B parameters per forward pass

Misconception: “Parameter count predicts latency linearly.”
Latency depends on memory bandwidth, not just FLOPs. For autoregressive generation, the bottleneck is loading weights from VRAM/HBM into compute units — a memory-bound operation. Doubling parameters roughly doubles latency if the model fits in the same memory tier. But crossing a VRAM boundary (e.g., 24 GB → 48 GB requiring multi-GPU or offloading) introduces nonlinear latency spikes from PCIe/NVLink transfer and kernel launch overhead.

Misconception: “You need the largest model for best results.”
Task-specific distillation, routing, and retrieval often beat raw parameter scaling. A 7B model with RAG over a curated corpus outperforms a 70B model with no retrieval on domain-specific QA. For many production workloads, a cascade — small model for classification/routing, larger model only for complex reasoning — delivers better cost/latency/quality tradeoffs than a single monolithic model. This is exactly the routing logic that gateways like n4n.ai implement: client directives steer requests to the appropriate model tier without application code changes.

Practical guidance for engineers

When evaluating models for a workload, start with the active parameter count (total for dense, routed for MoE), quantization level that fits your hardware, and context window requirements. Benchmark your specific task — generic benchmarks (MMLU, GSM8K) correlate poorly with domain performance. Measure:

• Time to first token (TTFT) at your typical prompt length
• Tokens per second (TPS) at your typical generation length
• VRAM usage with your batch size and KV cache configuration
• Quality on a held-out evaluation set representative of production inputs

Parameter count is a useful first filter, but it’s the beginning of the analysis, not the conclusion.