7B vs 70B vs 405B: what model size actually changes

Model size is the single lever that moves every other dial: memory footprint, token throughput, reasoning depth, and dollar cost per million tokens. The 7B vs 70B vs 405B decision isn’t about “better” — it’s about which constraints you’re willing to accept. This breakdown cuts through marketing claims and gives you the concrete dimensions that actually change when you cross parameter boundaries.

What model size actually means

Parameter count is a proxy for the model’s capacity to store and compose patterns from training data. A 7B model has roughly 7 billion weights; 405B has 405 billion. But the practical consequences cascade through your entire stack.

At inference time, the dominant memory consumer isn’t the weights — it’s the KV cache. For a 4k context window at FP16, a 7B model needs ~1.5 GB for weights plus ~0.5 GB for KV cache. A 405B model needs ~810 GB for weights plus ~40 GB for KV cache. That difference dictates whether you run on a single GPU, a node, or a cluster.

Quantization changes the arithmetic but not the hierarchy. A 4-bit 70B model fits in 48 GB VRAM (2× A100 40GB with tensor parallelism). A 4-bit 405B needs ~200 GB — still multi-node. The 7B class runs comfortably on consumer hardware; the 405B class does not.

Capabilities: where the step functions live

The capability curve isn’t linear. There are genuine step functions around 7B, 30–70B, and 400B+.

7B models (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B) handle single-turn QA, classification, extraction, and straightforward summarization. They fail at multi-step reasoning, long-context synthesis, and anything requiring world-model consistency across many tokens. They hallucinate more aggressively when pressed beyond their training distribution.

70B models (Llama-3.1-70B, Qwen2.5-72B, Nemotron-3-70B) cross the threshold for reliable multi-hop reasoning, code generation with context, and instruction following over 8k–128k tokens. They’re the smallest models that can reliably use tools in a ReAct loop without constant supervision. They still struggle with novel algorithmic problems and extended coherent generation (>4k tokens).

405B models (Llama-3.1-405B, Nemotron-3-405B) approach frontier proprietary models on benchmarks like MMLU-Pro, GPQA, and LiveCodeBench. They sustain coherence over 100k+ tokens, handle complex multi-file code tasks, and exhibit stronger “system 2” reasoning — they can backtrack, verify, and correct mid-generation. The gap to GPT-4o/Claude-3.5-Sonnet is real but narrowing; the gap to 70B is dramatic.

# Rough capability thresholds (qualitative, not benchmark scores)
CAPABILITY_THRESHOLDS = {
    "single_turn_qa": "7B",
    "classification_extraction": "7B",
    "multi_hop_reasoning": "70B",
    "tool_use_react": "70B",
    "long_context_synthesis_32k+": "70B",
    "multi_file_code_generation": "405B",
    "extended_coherent_generation_100k+": "405B",
    "novel_algorithmic_reasoning": "405B",
}

Price and cost model

Per-token pricing scales superlinearly with model size, but not proportionally to parameter count.

Provider API pricing reflects GPU-hour economics plus margin. Self-hosting flips the calculus: you pay for peak capacity, not per-token. A 70B model on 2× H100 serves ~2,000 tok/s sustained. At 80% utilization, that’s ~$0.004/M tokens — two orders of magnitude cheaper than API — but you eat the fixed cost regardless of traffic.

Hidden cost: context length. A 128k context request on 405B consumes ~1.2 GB KV cache per sequence. At 4-bit, that’s 300 MB. Ten concurrent 128k requests = 3 GB KV cache alone. On 7B, the same workload is ~150 MB. If your workload is long-context, the memory tax favors smaller models aggressively.

Latency and throughput

Latency has two components: time-to-first-token (TTFT) and inter-token latency (decode speed). Both scale with model size and sequence length.

TTFT is dominated by prompt processing (prefill), which is compute-bound and parallelizes well across GPUs. A 7B model prefill for 4k tokens takes ~50 ms on H100. 70B takes ~300 ms. 405B takes ~1.5 s. These numbers assume tensor parallelism across 8 GPUs; single-GPU prefill is 8× slower.

Decode speed is memory-bandwidth bound. Each generated token reads the full model weights plus KV cache from VRAM. On H100 (3 TB/s bandwidth):

• 7B 4-bit: ~15,000 tok/s theoretical, ~8,000 tok/s sustained
• 70B 4-bit: ~1,500 tok/s theoretical, ~800 tok/s sustained
• 405B FP8: ~300 tok/s theoretical, ~150 tok/s sustained

Batch inference changes the picture. With continuous batching (vLLM, TensorRT-LLM), you amortize weight reads across many sequences. A 70B model at batch 32 sustains ~25,000 tok/s aggregate throughput. But per-sequence latency stays roughly constant — you trade latency for throughput.

# vLLM config snippet showing model-size-dependent tuning
model: meta-llama/Llama-3.1-70B-Instruct
tensor_parallel_size: 4
gpu_memory_utilization: 0.9
max_num_batched_tokens: 8192
max_num_seqs: 256
# For 405B, you'd need:
# tensor_parallel_size: 8
# pipeline_parallel_size: 2
# max_num_batched_tokens: 4096

Ergonomics and deployment

7B: Runs on a single 24 GB GPU (RTX 3090/4090, A10G). Deploys as a single container. Cold start < 10 s. Fits in Kubernetes with resources.limits.nvidia.com/gpu: 1. No tensor parallelism needed. Quantization (AWQ, GPTQ, GGUF) is mature and lossless for most tasks.

70B: Minimum 2× 40 GB GPUs (A100 40GB, H100 80GB) for FP8/4-bit with tensor parallelism. Requires tensor_parallel_size=2 at minimum. Cold start 30–60 s (weight loading). Kubernetes needs pod anti-affinity, shared memory tuning, and possibly NCCL config. Quantization artifacts appear in long-form generation — FP8 or BF16 preferred for quality.

405B: Minimum 8× H100 80GB (640 GB VRAM) for FP8 tensor parallelism. Realistically 16–32 GPUs with pipeline parallelism for acceptable latency. Cold start 2–5 minutes. Requires dedicated node pools, InfiniBand/RoCE, NCCL tuning, and careful capacity planning. Not a “deploy and forget” model — it’s a cluster workload.

# 7B: single GPU, trivial
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct

# 70B: tensor parallel, 2 GPUs
docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 --dtype float8

# 405B: multi-node, not a docker run command
# Requires ray/skyPilot/k8s operator, IB fabric, 8+ H100s

Ecosystem and tooling

The 7B and 70B classes have deep quantization ecosystems: AWQ, GPTQ, GGUF, EXL2, HQQ. Every inference engine (vLLM, TGI, TensorRT-LLM, llama.cpp, Ollama) supports them natively. LoRA/QLoRA adapters are widely available. Fine-tuning recipes are battle-tested.

The 405B class is a different story. Quantization to 4-bit degrades quality noticeably on complex reasoning. FP8 is the practical floor. LoRA at 405B requires 8× H100 just for adapter training. Few open adapters exist. Fine-tuning requires full-sharded data parallelism (FSDP) or YaRN-style context extension — research-grade tooling, not production-ready.

Model routing matters here. If you run a gateway that routes by task complexity, you need reliable complexity signals. Most teams end up with heuristic classifiers (token count, keyword triggers, embedding similarity) that route 80% of traffic to 7B/70B and escalate the rest. n4n.ai handles this with client routing directives and automatic fallback when a provider degrades — but the routing logic itself is your responsibility.

Hard limits

Context window: All three classes now support 128k+ context (Llama 3.1, Qwen 2.5). But effective context differs. 7B models lose retrieval accuracy past ~16k tokens in needle-in-haystack tests. 70B holds to ~64k. 405B maintains fidelity at 128k. If your RAG pipeline stuffs 50k tokens into context, 7B will hallucinate citations.

Structured output: 7B models struggle with complex JSON schemas (nested objects, enums, oneOf). 70B handles most schemas reliably with constrained decoding (outlines, guidance, instructor). 405B is near-perfect but you pay 10× for the privilege.

Multilingual: 7B models are English-centric. 70B adds strong Spanish, French, Chinese, Japanese, Korean. 405B adds another 20+ languages with near-native fluency. If you serve global traffic, the threshold is 70B.

Tool calling: 7B models can emit tool calls but fail at parameter extraction and error recovery. 70B is the minimum for production ReAct loops. 405B handles parallel tool calls, recursive correction, and complex API composition.

Comparison table

Which to choose

Choose 7B when:

• Latency budget < 200 ms p99
• Traffic volume > 10k req/min (cost dominates)
• Tasks are classification, extraction, single-turn QA, routing, intent detection
• You run on consumer GPUs or spot instances
• You need horizontal scaling without cluster coordination

Choose 70B when:

• You need reliable tool use, multi-hop reasoning, or 32k+ context
• Code generation spans multiple files or requires repository context
• Structured output schemas are non-trivial
• You have 2–8 H100/A100s available (or budget for API)
• 80% of your traffic is “hard” but 20% is “easy” — pair with a 7B router

Choose 405B when:

• You’re building a flagship product where model quality is the differentiator
• Tasks require 100k+ context synthesis (legal, financial, scientific literature)
• Novel algorithmic reasoning or multi-step planning is core to the product
• You have dedicated ML infra team and 16+ H100s
• You can amortize fixed cost over sustained high-value traffic

The pragmatic pattern most teams converge on: Route 70% to 7B, 25% to 70B, 5% to 405B. Implement a lightweight classifier (fastText, mini-BERT, or heuristic) that inspects the prompt and routes accordingly. Cache aggressively at the 7B layer. Escalate on low confidence, long context, or explicit user tier. This gives you 7B economics for the bulk, 70B quality for the meat, and 405B ceiling for the edge cases — without provisioning a 405B cluster for your median request.