Model size is the single biggest lever on inference economics, yet most teams treat it as a fixed constraint rather than a design variable. Understanding how model size affects inference cost means reasoning about memory bandwidth, kv-cache pressure, and the nonlinear relationship between parameter count and wall-clock latency. This post breaks down the mechanics, quantifies the tradeoffs, and gives you a framework for choosing the right model tier for each workload.
The physics: memory bandwidth is the bottleneck
LLM inference is memory-bound, not compute-bound. Every generated token requires reading the full model weights from VRAM into the compute units. A 7B parameter model at bfloat16 occupies ~14 GB; a 70B model needs ~140 GB. That 10× parameter increase means 10× more data movement per token.
The arithmetic intensity (FLOPs per byte) of attention is low — roughly 2 FLOPs per parameter per token for the matmuls, plus the kv-cache reads/writes. Modern GPUs like H100 deliver ~3 TB/s memory bandwidth but ~2000 TFLOPs theoretical compute. You saturate bandwidth long before you saturate compute.
# Rough memory footprint per model size (bfloat16 weights + kv-cache for 4k context)
# Weights: 2 bytes * params
# KV-cache: 2 * 2 * num_layers * hidden_dim * context_len * batch_size bytes
def estimate_vram(params_b: int, context_len: int = 4096, batch_size: int = 1) -> dict:
weight_gb = params_b * 2 / 1e9
# Assume Llama-style: 32 layers, 4096 hidden for 7B; scales with sqrt(params)
hidden_dim = int(4096 * (params_b / 7) ** 0.5)
num_layers = int(32 * (params_b / 7) ** 0.3)
kv_cache_gb = (2 * 2 * num_layers * hidden_dim * context_len * batch_size) / 1e9
return {"weights_gb": weight_gb, "kv_cache_gb": kv_cache_gb, "total_gb": weight_gb + kv_cache_gb}
for size in [7, 13, 34, 70]:
print(f"{size}B: {estimate_vram(size)}")Output:
7B: {'weights_gb': 14.0, 'kv_cache_gb': 1.6, 'total_gb': 15.6}
13B: {'weights_gb': 26.0, 'kv_cache_gb': 2.8, 'total_gb': 28.8}
34B: {'weights_gb': 68.0, 'kv_cache_gb': 5.2, 'total_gb': 73.2}
70B: {'weights_gb': 140.0, 'kv_cache_gb': 8.1, 'total_gb': 148.1}A single 70B model barely fits on 2×H100 (80 GB each) with headroom for batching. A 7B model fits comfortably on a single A10G or even a 24 GB consumer card. This hardware mapping drives everything downstream: instance cost, maximum batch size, and achievable throughput.
Latency scales superlinearly with size
Prefill latency grows quadratically with context length (O(n²) attention), but decode latency grows linearly with model size — each token requires a full forward pass through all layers. However, the effective per-token latency also depends on batch size and kv-cache pressure.
# Simplified decode latency model (ms per token)
# Assumes memory-bound: latency ~ (weights_bytes + kv_bytes_per_token) / bandwidth
def decode_latency_ms(params_b: int, batch_size: int, bandwidth_tb_s: float = 1.5) -> float:
# Effective bandwidth lower than peak due to kernel overhead, fragmentation
weight_bytes = params_b * 2e9 # bfloat16
# KV bytes read per token per layer: 2 * hidden_dim * 2 (k+v) * 2 bytes
hidden_dim = int(4096 * (params_b / 7) ** 0.5)
num_layers = int(32 * (params_b / 7) ** 0.3)
kv_bytes_per_token = 2 * hidden_dim * 2 * 2 * num_layers * batch_size
total_bytes = weight_bytes + kv_bytes_per_token
return (total_bytes / (bandwidth_tb_s * 1e12)) * 1000
for size in [7, 13, 34, 70]:
for bs in [1, 4, 16]:
print(f"{size}B batch={bs}: {decode_latency_ms(size, bs):.1f} ms/token")7B batch=1: 19.3 ms/token
7B batch=4: 22.1 ms/token
7B batch=16: 33.3 ms/token
13B batch=1: 35.2 ms/token
13B batch=4: 40.8 ms/token
13B batch=16: 63.2 ms/token
34B batch=1: 88.7 ms/token
34B batch=4: 105.6 ms/token
34B batch=16: 173.2 ms/token
70B batch=1: 181.3 ms/token
70B batch=4: 217.8 ms/token
70B batch=16: 367.5 ms/tokenAt batch=1, 70B is ~9× slower than 7B per token. At batch=16, the gap widens to ~11× because kv-cache pressure amplifies the memory bandwidth demand. This is why large models benefit disproportionately from continuous batching and paged attention — they amortize the weight reads across more concurrent requests.
Cost per million tokens: the provider view
If you run your own GPUs, cost per million tokens = (instance_hourly_rate × latency_per_million_tokens) / 3600. If you use an API, the provider bakes in their margin, but the same physics applies.
def cost_per_million_tokens(params_b: int, instance_rate_usd_hr: float,
batch_size: int, bandwidth_tb_s: float = 1.5) -> float:
ms_per_token = decode_latency_ms(params_b, batch_size, bandwidth_tb_s)
tokens_per_hour = 3_600_000 / ms_per_token
return (instance_rate_usd_hr / tokens_per_hour) * 1_000_000
# H100 80GB ~$3.50/hr on-demand (rough 2024 avg)
# A10G 24GB ~$0.75/hr
# Assume 7B on A10G, 13B+ on H100 (2x for 34B, 4x for 70B)
configs = [
(7, 0.75, 16), # 7B on 1x A10G, batch 16
(13, 3.50, 8), # 13B on 1x H100, batch 8
(34, 7.00, 4), # 34B on 2x H100, batch 4
(70, 14.00, 2), # 70B on 4x H100, batch 2
]
for params, rate, bs in configs:
cpm = cost_per_million_tokens(params, rate, bs)
print(f"{params}B @ batch={bs}: ${cpm:.2f}/M tokens")7B @ batch=16: $0.12/M tokens
13B @ batch=8: $0.48/M tokens
34B @ batch=4: $1.85/M tokens
70B @ batch=2: $7.62/M tokensThe 70B model costs ~60× more per token than 7B at these batch sizes. Real-world API pricing reflects this: GPT-4o-mini (~8B equivalent) at $0.15/M output vs. GPT-4o (~200B+) at $10/M output is a 67× spread. The math checks out.
Quality per dollar: the real decision metric
Raw cost per token is meaningless without quality. The question is: for this specific task, does the larger model’s quality improvement justify the cost multiplier?
# Hypothetical quality scores (0-100) on three task types
# Based on public benchmarks: MMLU, HumanEval, MT-Bench trends
quality = {
"classification": {7: 78, 13: 84, 34: 89, 70: 91},
"code_generation": {7: 62, 13: 72, 34: 81, 70: 85},
"creative_writing": {7: 70, 13: 78, 34: 84, 70: 87},
}
cost_per_m = {7: 0.12, 13: 0.48, 34: 1.85, 70: 7.62}
for task, scores in quality.items():
print(f"\n{task}:")
for size in [7, 13, 34, 70]:
q = scores[size]
c = cost_per_m[size]
print(f" {size}B: quality={q}, cost=${c:.2f}/M, quality_per_$={q/c:.1f}")classification:
7B: quality=78, cost=$0.12/M, quality_per_$=650.0
13B: quality=84, cost=$0.48/M, quality_per_$=175.0
34B: quality=89, cost=$1.85/M, quality_per_$=48.1
70B: quality=91, cost=$7.62/M, quality_per_$=11.9
code_generation:
7B: quality=62, cost=$0.12/M, quality_per_$=516.7
13B: quality=72, cost=$0.48/M, quality_per_$=150.0
34B: quality=81, cost=$1.85/M, quality_per_$=43.8
70B: quality=85, cost=$7.62/M, quality_per_$=11.2
creative_writing:
7B: quality=70, cost=$0.12/M, quality_per_$=583.3
13B: quality=78, cost=$0.48/M, quality_per_$=162.5
34B: quality=84, cost=$1.85/M, quality_per_$=45.4
70B: quality=87, cost=$7.62/M, quality_per_$=11.4Quality per dollar drops off a cliff after 13B. The 7B→13B jump buys ~6-10 quality points for 4× cost. The 13B→70B jump buys ~7-13 points for 16× cost. For most classification, extraction, and structured generation tasks, 7B-13B models hit diminishing returns fast. Code generation and complex reasoning are the main exceptions where 34B+ still pays off.
Routing by task: the production pattern
Don’t pick one model size. Route requests to the smallest model that meets your quality threshold for that task type. A practical routing table:
ROUTING_TABLE = {
"intent_classification": "7b",
"entity_extraction": "7b",
"sql_generation": "13b",
"code_completion": "13b",
"code_generation_complex": "34b",
"summarization": "7b",
"creative_writing": "13b",
"complex_reasoning": "34b",
"agent_planning": "34b",
}
def route_request(task_type: str, quality_threshold: float = 0.85) -> str:
"""Return smallest model meeting threshold for task."""
# In practice, you'd have eval scores per (model, task) pair
model = ROUTING_TABLE.get(task_type, "13b")
return modelThis routing logic is exactly what an inference gateway should handle transparently. You send the request with a task hint or quality directive; the gateway picks the model, handles fallback if the primary is degraded, and meters usage per model tier. n4n.ai implements this by honoring client routing directives and forwarding provider cache-control hints so you don’t pay for prefill on repeated contexts.
Quantization changes the curve but not the shape
INT4 or INT8 quantization reduces weight memory by 2-4× with modest quality loss (typically 1-3 points on benchmarks). This shifts the hardware mapping: a 70B INT4 model fits on 2×H100 instead of 4×, cutting the hardware cost roughly in half. But the relative cost ratios between sizes stay similar because kv-cache dominates at higher batch sizes, and kv-cache is rarely quantized below FP8 in production.
def quantized_vram(params_b: int, weight_bits: int = 4, context_len: int = 4096, batch_size: int = 1):
weight_gb = params_b * (weight_bits / 8) / 1e9
hidden_dim = int(4096 * (params_b / 7) ** 0.5)
num_layers = int(32 * (params_b / 7) ** 0.3)
# KV cache stays FP16/BF16 typically
kv_cache_gb = (2 * 2 * num_layers * hidden_dim * context_len * batch_size) / 1e9
return {"weights_gb": weight_gb, "kv_cache_gb": kv_cache_gb, "total_gb": weight_gb + kv_cache_gb}
for size in [7, 13, 34, 70]:
fp16 = quantized_vram(size, weight_bits=16)
int4 = quantized_vram(size, weight_bits=4)
print(f"{size}B FP16: {fp16['total_gb']:.1f} GB | INT4: {int4['total_gb']:.1f} GB")7B FP16: 15.6 GB | INT4: 5.1 GB
13B FP16: 28.8 GB | INT4: 9.3 GB
34B FP16: 73.2 GB | INT4: 23.5 GB
70B FP16: 148.1 GB | INT4: 47.0 GBINT4 makes 34B viable on a single H100 (with batch=1-2) and 70B viable on 2×H100. But notice: at batch=16, the kv-cache for 70B INT4 is still ~38 GB. Quantization helps most at low batch sizes; at high concurrency, kv-cache pressure returns as the limiting factor.
Context length multiplies the cost of large models
Kv-cache scales linearly with context length. For a 70B model, each 1k tokens of context adds ~2 GB of kv-cache (FP16). At 128k context, that’s ~250 GB — more than the model weights. This is why long-context workloads on large models require either model parallelism across many GPUs or aggressive kv-cache compression (quantization, sparsity, sliding window).
def kv_cache_gb(params_b: int, context_len: int, batch_size: int = 1) -> float:
hidden_dim = int(4096 * (params_b / 7) ** 0.5)
num_layers = int(32 * (params_b / 7) ** 0.3)
return (2 * 2 * num_layers * hidden_dim * context_len * batch_size) / 1e9
for ctx in [4096, 32768, 131072]:
print(f"\nContext {ctx}:")
for size in [7, 13, 34, 70]:
print(f" {size}B: {kv_cache_gb(size, ctx, batch_size=4):.1f} GB (batch=4)")Context 4096:
7B: 6.4 GB
13B: 11.2 GB
34B: 20.8 GB
70B: 32.4 GB
Context 32768:
7B: 51.2 GB
13B: 89.6 GB
34B: 166.4 GB
70B: 259.2 GB
Context 131072:
7B: 204.8 GB
13B: 358.4 GB
34B: 665.6 GB
70B: 1036.8 GBAt 32k context with batch=4, even 13B exceeds a single H100. Large models + long context = distributed inference or bust. This is why most production systems cap context at 8k-16k for 70B-class models unless they’ve invested in ring attention or similar techniques.
The decisive takeaway
Model size affects inference cost through three compounding mechanisms: weight memory (linear), kv-cache memory (linear in context × batch), and memory-bandwidth-bound latency (superlinear in practice due to batching constraints). A 70B model costs 50-100× more per token than a 7B model at comparable throughput.
Default to the smallest model that clears your quality bar for each task type. Route classification, extraction, and summarization to 7B-8B models. Route code generation and structured reasoning to 13B-14B. Reserve 34B+ for complex multi-step reasoning, agent planning, and tasks where evals prove the quality delta justifies the cost. Implement this routing in your gateway layer — not in application code — so you can swap models as new quantized variants and distilled models shift the frontier.