LLM parameters explained: what a "70B" model means

When you see “70B” in a model name like Llama 3 70B or Nemotron 3 Ultra 70B, it refers to the total number of trainable parameters — approximately 70 billion weights — that define the model’s learned behavior. Each parameter is a floating-point value (typically 16-bit or 4-bit quantized) adjusted during training to minimize loss on the training corpus. The parameter count determines the model’s capacity to represent complex functions, but it also directly dictates VRAM requirements, inference latency, and cost.

How parameter counts work

A transformer model’s parameters live in three main places: token embeddings, attention layers, and feed-forward networks (FFNs). For a standard decoder-only architecture like Llama 3, the parameter budget breaks down roughly as follows:

• Token embeddings: vocab_size × hidden_dim. Llama 3 uses a 128K vocabulary and 8,192 hidden dimensions for the 70B model, consuming ~1B parameters.
• Attention projections: Each layer has Q, K, V, and O projection matrices. With 80 layers, 8,192 hidden dim, and 8,192 attention dim (128 heads × 64), this accounts for ~4B parameters per layer × 80 = ~32B.
• FFN layers: Each layer uses a SwiGLU feed-forward network with an intermediate dimension of 28,672. That’s 2 × hidden_dim × intermediate_dim per layer ≈ 470M per layer × 80 = ~37.6B.
• Layer norms and output head: Negligible (< 0.5B total).

# Rough parameter accounting for Llama 3 70B
vocab_size = 128_256
hidden_dim = 8_192
num_layers = 80
num_heads = 128
head_dim = 64
intermediate_dim = 28_672

embed_params = vocab_size * hidden_dim                    # ~1.05B
attn_params_per_layer = 4 * hidden_dim * (num_heads * head_dim)  # ~4.03B
ffn_params_per_layer = 2 * hidden_dim * intermediate_dim         # ~469.8M
total = embed_params + num_layers * (attn_params_per_layer + ffn_params_per_layer)
print(f"Total: {total / 1e9:.1f}B")  # ~70.6B

The “B” suffix rounds to the nearest billion. A “7B” model might be 6.7B or 7.3B; “70B” typically lands between 67B and 72B depending on architecture choices like grouped-query attention or tied embeddings.

Why parameter count matters

Parameter count is the primary knob for three engineering constraints: memory footprint, compute per token, and model capability.

Memory footprint

At inference time, you need to store model weights plus the KV cache for each active sequence. For a 70B model at different precisions:

The KV cache scales with 2 × num_layers × hidden_dim × context_length × batch_size × bytes_per_element. For 70B at 4K context with BF16: 2 × 80 × 8192 × 4096 × 1 × 2 ≈ 1.07 GB. At 128K context, that jumps to ~34 GB — often larger than the quantized weights themselves.

def estimate_kv_cache_gb(num_layers, hidden_dim, context_len, batch_size=1, dtype_bytes=2):
    # 2 tensors (K and V) per layer
    return (2 * num_layers * hidden_dim * context_len * batch_size * dtype_bytes) / 1e9

print(f"4K ctx: {estimate_kv_cache_gb(80, 8192, 4096):.1f} GB")
print(f"128K ctx: {estimate_kv_cache_gb(80, 8192, 131072):.1f} GB")

This is why 70B models at FP16 require multi-GPU setups (2× H100 80GB or 4× A100 40GB), while INT4 fits on a single H100 or dual 3090/4090 consumer cards.

Compute per token

FLOPs per forward pass scale roughly as 6 × params × tokens for dense transformers (2 FLOPs per multiply-add × 3 for QKV + FFN). For 70B generating one token:

6 × 70B × 1 = 420 GFLOPs per token

At 50 tokens/sec, that’s 21 TFLOPs sustained. An H100 delivers ~990 TFLOPs BF16 dense, so theoretical max throughput is ~47 tokens/sec per GPU — before memory bandwidth and kernel overhead. Real-world vLLM or TGI deployments typically see 30–40 tok/s on H100 for 70B INT4.

Smaller models scale linearly: 8B is ~8.7× faster and uses ~8.7× less memory. This is why many production systems route simple queries to 8B models and escalate to 70B only when needed.

Capability scaling

Parameter count correlates with downstream performance, but the relationship follows diminishing returns. The Chinchilla scaling laws suggest optimal training tokens scale linearly with parameters: a 70B model needs ~1.4T tokens for compute-optimal training. Llama 3 70B trained on 15T tokens is significantly overtrained relative to Chinchilla, which explains its strong benchmarks despite “only” 70B parameters.

{
  "model": "Llama 3 70B",
  "params": 70_000_000_000,
  "training_tokens": 15_000_000_000_000,
  "tokens_per_param": 214,
  "chinchilla_optimal_tokens": 1_400_000_000_000,
  "overtraining_factor": 10.7
}

Overtraining smaller models (like Llama 3 8B at 15T tokens) narrows the gap with larger models on many tasks, but 70B still wins on complex reasoning, long-context retrieval, and multilingual consistency.

Concrete example: Llama 3 8B vs 70B in production

Consider a RAG pipeline answering technical questions from documentation. You have two deployment options:

Option A: Llama 3 8B INT4 on 1× A10G (24 GB VRAM)

• Model weights: ~4.7 GB
• KV cache (4K ctx): ~0.15 GB
• Headroom for batching: ~19 GB → batch size ~32 at 4K context
• Throughput: ~120 tok/s
• Cost: ~$0.75/hr (spot)

Option B: Llama 3 70B INT4 on 1× H100 (80 GB VRAM)

• Model weights: ~35 GB
• KV cache (4K ctx): ~1.3 GB
• Headroom: ~43 GB → batch size ~8 at 4K context
• Throughput: ~35 tok/s
• Cost: ~$2.50/hr (spot)

For a workload of 1,000 requests/day averaging 500 output tokens:

• Option A: 500K tokens / 120 tok/s ≈ 1.15 GPU-hours → $0.86/day
• Option B: 500K tokens / 35 tok/s ≈ 4 GPU-hours → $10/day

The 70B model costs ~12× more per token. Is it worth it? Run an eval:

# Pseudo-eval harness
questions = load_benchmark("tech-qa-v2")
results_8b = evaluate(model="llama3-8b-int4", questions=questions)
results_70b = evaluate(model="llama3-70b-int4", questions=questions)

print(f"8B accuracy: {results_8b.accuracy:.1%}")
print(f"70B accuracy: {results_70b.accuracy:.1%}")
print(f"Delta: {results_70b.accuracy - results_8b.accuracy:.1%}")

Typical results on technical QA: 8B scores ~78%, 70B scores ~87%. That 9% gap might justify the cost if errors are expensive (e.g., medical, legal, infrastructure code). For general chat or summarization, 8B often suffices.

A common pattern: route by query complexity. Use a lightweight classifier or heuristic (token count, domain keywords, required reasoning depth) to send 80% of traffic to 8B and 20% to 70B. This keeps average cost near the 8B baseline while capturing 70B quality where it matters.

Common misconceptions

“More parameters = smarter model”

Not necessarily. Architecture, training data quality, tokenization, and post-training (SFT, RLHF, DPO) matter as much as raw parameter count. Nemotron 3 Ultra 70B outperforms Llama 3 70B on many benchmarks despite identical parameter counts, due to better data curation and alignment. Conversely, a poorly trained 70B model can underperform a well-trained 30B model (e.g., early Falcon 70B vs. Qwen 1.5 32B).

“Quantization to 4-bit loses negligible quality”

For 70B models, INT4 with GPTQ or AWQ typically loses 1–2% absolute on MMLU and GSM8K. For 8B and smaller models, the gap widens to 3–5% because smaller models have less redundancy. Always eval your specific task after quantization. Dynamic per-channel scaling (AWQ) generally preserves more quality than static GPTQ for 70B.

“Context length is free”

KV cache grows linearly with context. A 70B model at 128K context needs ~34 GB KV cache alone (BF16). That’s 42% of an H100’s VRAM before weights. Techniques like KV cache quantization (KV8, FP8), sliding window attention, or MLA (DeepSeek-V2) mitigate this, but standard Llama 3 architectures pay the full cost. If you need 128K context on 70B, plan for 2× H100 or model parallelism.

“Parameter count determines licensing”

Parameter count has no legal bearing. Llama 3 8B and 70B share the same Llama 3 Community License. Mistral 7B and Mixtral 8×7B (47B active params) use Apache 2.0. Always check the specific model card, not the size.

Practical implications for engineers

Model selection checklist:

1. Eval first: Run your task-specific benchmark on 8B, 70B, and any intermediate sizes (34B, 40B) available. Don’t assume scaling curves.
2. Quantization target: INT4 is the default for 70B deployment. FP8 on H100 offers better quality with similar memory to INT4 but requires Hopper.
3. Batching strategy: 70B benefits more from continuous batching (vLLM, TGI) than smaller models because memory headroom is tighter. Set max_num_seqs to saturate VRAM without OOM.
4. Routing logic: Implement a router — even a simple keyword/length heuristic — to avoid burning 70B compute on “hello world” queries.
5. Observability: Track tokens/sec, TTFT, and error rates per model size. 70B deployments fail differently (OOM, timeout) than 8B.

When to default to 70B:

• Complex multi-step reasoning (code generation with dependencies, math proofs)
• Long-context synthesis (>32K tokens where 8B loses coherence)
• Multilingual tasks requiring low-resource language competence
• When you need a single model to cover diverse tasks without routing infrastructure

When 8B (or 3B) is sufficient:

• Classification, extraction, summarization of short docs
• Chat with constrained domains (FAQ, support)
• High-volume, latency-sensitive paths
• Edge or on-device deployment

The parameter count in the model name is a proxy for capacity, cost, and capability — not a guarantee of any of them. Treat it as a starting constraint, not a quality certificate.