How many parameters does GPT-4 have? What's known

The question of how many parameters does GPT-4 have has generated endless speculation since its release in March 2023. OpenAI has never disclosed the figure, breaking from their practice with GPT-3 (175B parameters) and GPT-2 (1.5B). The silence is intentional. Parameter count has become a poor proxy for capability, and the industry’s fixation on it obscures more important architectural decisions. This post examines what we can actually infer, why the number matters less than the training dynamics, and what engineers should actually care about when selecting models.

The official position and why it changed

OpenAI’s technical report for GPT-4 contains no parameter count. Sam Altman confirmed in a 2023 interview that they would not disclose it, stating “parameter count is not the right way to think about model quality.” This represents a deliberate shift. With GPT-3, the 175B figure served as a marketing anchor — a concrete claim of scale that differentiated from competitors. By GPT-4, the competitive landscape had shifted. Multiple open models (LLaMA 65B, Falcon 180B) exceeded or approached GPT-3’s scale without matching its capabilities. Parameter count had decoupled from performance.

The decision also reflects competitive opacity. Disclosing architecture details — parameter count, layer count, attention head configuration — provides signal to competitors about training compute budgets and architectural choices. In a market where training runs cost tens of millions, that signal has value.

What the scaling laws actually tell us

Chinchilla scaling laws (Hoffmann et al., 2022) established that optimal training allocates compute between parameters and tokens such that both scale proportionally. For a given compute budget C, optimal parameters N and tokens D follow:

N ∝ C^0.5
D ∝ C^0.5

GPT-3 (175B params, ~300B tokens) was undertrained relative to Chinchilla-optimal — it should have seen ~3.5T tokens. If GPT-4 followed Chinchilla scaling with a 10x compute increase over GPT-3, we’d expect roughly 500B-1T parameters trained on 3-5T tokens. But this assumes dense architecture and unchanged training objectives.

The Chinchilla paper itself notes that these laws describe dense transformers trained with standard next-token prediction. Mixture-of-experts (MoE), multimodal objectives, and reinforcement learning from human feedback (RLHF) all break the assumptions. Any parameter estimate derived purely from scaling laws carries massive uncertainty.

Mixture-of-experts changes the math entirely

The most credible architectural leak comes from a semi-credible source: a 2023 tweet by George Hotz claiming GPT-4 uses 16 experts with ~111B parameters each, totalling ~1.8T parameters but only ~280B active per forward pass. While unverified, this aligns with several independent signals:

1. Inference latency profiles — GPT-4’s latency per token is consistent with MoE models where only a fraction of parameters activate
2. OpenAI’s hiring — They recruited heavily from the MoE research community (including authors of the Switch Transformer and GLaM papers) in 2021-2022
3. Industry convergence — Google’s GLaM (1.2T total, 96B active), DeepMind’s Gopher (280B dense), and NVIDIA’s Megatron-MoE all demonstrated MoE’s compute efficiency

If GPT-4 is MoE, the question “how many parameters does GPT-4 have” has two answers: total parameters (likely 1T-2T) and active parameters (likely 200B-400B). The latter determines inference cost and latency; the former determines training compute and memorization capacity.

# Conceptual MoE forward pass — only 2 of 16 experts activate per token
class MoELayer(nn.Module):
    def __init__(self, num_experts=16, expert_capacity=2, d_model=4096):
        super().__init__()
        self.experts = nn.ModuleList([
            Expert(d_model) for _ in range(num_experts)
        ])
        self.router = nn.Linear(d_model, num_experts)
        self.expert_capacity = expert_capacity
    
    def forward(self, x):
        # x: [batch, seq_len, d_model]
        logits = self.router(x)  # [batch, seq_len, num_experts]
        gates, indices = torch.topk(logits, self.expert_capacity, dim=-1)
        gates = F.softmax(gates, dim=-1)
        
        # Sparse dispatch — only selected experts compute
        out = torch.zeros_like(x)
        for k in range(self.expert_capacity):
            expert_idx = indices[..., k]
            gate = gates[..., k:k+1]
            expert_out = self._dispatch_to_experts(x, expert_idx)
            out += gate * expert_out
        return out

This architecture explains GPT-4’s ability to handle 32K and 128K context windows — the active parameter count stays manageable even as total capacity grows.

Multimodality adds parameters that don’t count the same way

GPT-4 accepts image inputs. The vision encoder (likely a ViT variant) adds parameters that process pixels, not tokens. These parameters don’t participate in language modeling the same way. A 300M-parameter ViT-L/14 encoder contributes to total parameter count but not to the “language model size” in any meaningful sense.

Furthermore, the cross-modal alignment — projecting vision embeddings into the language model’s residual stream — adds projection layers and potentially cross-attention layers. These are parameter-expensive but token-cheap at inference (images become a fixed token prefix).

Engineers should treat multimodal parameter counts as a separate budget. The language reasoning capacity correlates with text-active parameters, not total parameters including vision encoders.

What we can infer from inference economics

OpenAI’s pricing provides indirect evidence. GPT-4-8K launched at $0.03/1K input tokens, $0.06/1K output. GPT-3.5-turbo was $0.0015/$0.002 — a 20x premium. By late 2024, GPT-4o dropped to $2.50/$10 per million tokens (~5x over 4o-mini).

Inference cost scales roughly linearly with active parameters (plus KV cache for context). A 20x price premium suggests 10-20x active parameters over GPT-3.5-turbo (~20B active if MoE, or ~175B if dense). This points to 200B-400B active parameters — consistent with the MoE hypothesis.

But pricing also reflects:

• KV cache memory (scales with context length and layers)
• Expert routing overhead (MoE adds all-to-all communication)
• Monopoly pricing power (OpenAI charges what the market bears)
• Subsidized smaller models (4o-mini may run at loss leader pricing)

Pricing is a noisy signal. Don’t overfit.

Why parameter count is the wrong metric for engineers

The industry’s obsession with parameter count stems from a time when architecture was homogeneous (dense decoder-only transformers) and training objectives were uniform (next-token prediction). That era ended around 2022. Today, the following matter more:

Training token count and quality. LLaMA-2 70B trained on 2T tokens outperforms many 100B+ models trained on less data. The Pile, RedPajama, FineWeb — data curation now drives more variance than parameter scaling.

Architecture variants. MoE, grouped-query attention, sliding window attention, RoPE vs ALiBi, parallel vs sequential attention+MLP — these choices change the FLOPs/parameter ratio and the quality/FLOP ratio.

Post-training. RLHF, DPO, constitutional AI, RLAIF, synthetic data distillation — a 7B model with excellent post-training beats a 70B base model on instruction following.

Inference optimization. Quantization (AWQ, GPTQ, GGUF), speculative decoding, continuous batching, PagedAttention — these determine deployed latency and cost, not raw parameter count.

# Example: Same model, different quantization, vastly different VRAM
# Llama-3-70B-Instruct
# FP16:     ~140 GB VRAM (7 x H100-80GB)
# INT4 AWQ: ~38 GB VRAM  (1 x H100-80GB with headroom)
# INT4 GGUF: ~38 GB RAM  (runs on MacBook Pro M3 Max)

An engineer choosing a model should benchmark their workload on their infrastructure with their quantization. Parameter count is a starting heuristic, not a decision criterion.

The MoE active parameter rule of thumb

If you must estimate: for MoE models, active parameters ≈ total_parameters / num_experts * experts_per_token. Typical configurations:

Active parameters determine:

• FLOPs per token (≈ 2 × active_params for forward pass)
• KV cache size (scales with layers × hidden_dim, not total experts)
• Model parallelism strategy (expert parallelism vs tensor parallelism)

Total parameters determine:

• Training compute (all experts see gradient updates)
• Memorization capacity (rare facts distributed across experts)
• Checkpoint size and storage

What GPT-4’s successors tell us

GPT-4o and o1 provide additional clues. GPT-4o’s latency improvement (sub-100ms first token vs 500ms+ for GPT-4) suggests architectural changes: possibly fewer layers, grouped-query attention, or a smaller active parameter count with better token efficiency. o1’s reasoning tokens imply a different inference-time compute scaling — test-time compute replaces parameter count as the quality lever.

The trajectory is clear: parameter count is decoupling from capability entirely. A 3B model with search, tools, and test-time compute (like o1-mini) outperforms a 70B model without them on reasoning tasks. The industry is moving toward “systems, not models” — where the parameter count of the base LLM is one component among many.

Practical guidance for model selection

Stop asking how many parameters does GPT-4 have. Start asking:

1. What’s the active parameter count at inference? Determines your GPU budget.
2. What’s the context window and KV cache cost? Determines your max batch size.
3. Does it support your required modalities? Vision, audio, function calling.
4. What’s the licensing? Apache 2.0 vs custom vs API-only.
5. How does it benchmark on your eval set? Not MMLU, not GSM8K — your task.

# A real evaluation harness beats any parameter heuristic
def evaluate_model(model, eval_dataset, metrics):
    results = {}
    for task_name, dataset in eval_dataset.items():
        predictions = []
        for example in dataset:
            pred = model.generate(example.input, max_tokens=512)
            predictions.append(pred)
        results[task_name] = {
            metric: metric_fn(predictions, dataset.targets)
            for metric in metrics
        }
    return results

# Run this on your shortlist. Parameter count doesn't appear.

If you’re routing through a gateway that abstracts providers (like n4n.ai), you can A/B test across model families without rewriting integration code. The gateway handles the OpenAI-compatible endpoint normalization, fallback logic, and usage metering — you just compare results.

The decisive takeaway

GPT-4’s parameter count is undisclosed because it’s the wrong question. The credible evidence points to a mixture-of-experts architecture with roughly 1.5-2 trillion total parameters and 200-400 billion active per forward pass. But that number explains little about why GPT-4 performs as it does.

The engineering reality: data quality, post-training, inference optimization, and system design now dominate parameter count as performance levers. A 7B model with RAG, tool use, and careful prompting solves more production problems than a 70B model without them. Parameter count is a legacy metric from the dense-model era. Treat it as historical context, not a design input.

Build evals. Measure latency on your hardware. Optimize the system around the model. That’s where the leverage lives.