Small models, big throughput: the case for sub-10B LLMs

Most teams reach for a 70B-parameter model by default, then wonder why their inference bill explodes and p99 latency climbs. For the majority of production workloads—classification, extraction, templated generation—a sub-10b llm throughput advantage of 5–10x makes smaller models the fiscally responsible choice, with negligible quality loss. The thesis here is simple: start small, measure, and escalate only when the task demands it.

The physics of inference: why size kills throughput

Transformer decoding is memory-bandwidth bound, not compute bound, for the generate phase. Each token requires reading the full set of weights from HBM into compute units. A 7B model in FP16 occupies ~14 GB; a 70B occupies ~140 GB. Even on identical accelerator hardware, the 70B forces roughly 10x the memory traffic per token, which directly caps sub-10b llm throughput at a fraction of what a small model sustains.

Bandwidth-bound math

# First-order bandwidth-bound estimate (ignoring KV cache)
def max_tokens_per_sec(params_b, bw_tbs, bits=16):
    bytes_per_param = bits / 8
    weight_bytes = params_b * 1e9 * bytes_per_param
    return bw_tbs * 1e12 / weight_bytes

# A100 80GB PCIe ~2 TB/s effective
print(max_tokens_per_sec(7, 2))   # ~143k tok/s theoretical per chip
print(max_tokens_per_sec(70, 2))  # ~14k tok/s

Reality is worse for large models because KV cache grows with batch size and context. A 7B model with 4k context uses ~0.5 GB KV per stream in FP16; a 70B uses ~5 GB. That difference dictates max concurrent batch size on a given GPU. The inverse relationship between parameter count and sub-10b llm throughput holds before you even consider quantization.

What sub-10B models are actually good at

Engineers underestimate how far a 7B or 3B model goes after instruction tuning or light LoRA adaptation. Tasks that are narrow, well-specified, and verifiable are ideal:

• Intent classification (support tickets, survey responses)
• PII redaction or JSON field extraction from documents
• Short-form rewriting, tone adjustment, summarization of small chunks
• Code completion for boilerplate or test scaffolding
• Draft generation that a human or larger model later refines

A typical extraction call against an OpenAI-compatible endpoint:

import json
from openai import OpenAI

client = OpenAI(base_url="https://inference.example.com/v1", api_key="key")

prompt = """Extract name and amount from invoice text as JSON.
Text: Invoice to Jane Doe for $420.00.
"""
r = client.chat.completions.create(
    model="qwen2.5-7b-instruct",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"},
    max_tokens=64,
)
print(json.loads(r.choices[0].message.content))

On a single L4 GPU, this 7B model serves ~1,200 requests/min at <200 ms p50. A 70B on the same hardware would need tensor parallelism and deliver maybe 120 req/min. That 10x sub-10b llm throughput difference is the gap between one GPU and a full rack.

Quantization and batching: squeezing more out

Post-training quantization to INT4 cuts weight footprint 4x, letting you fit a 7B model in 6 GB and batch larger. Combined with continuous batching (vLLM, TensorRT-LLM), you turn sporadic requests into high GPU utilization.

Serving with vLLM

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3-awq \
  --quantization awq \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --enable-prefix-caching

Prefix caching means repeated system prompts are computed once. For high-volume workloads with shared instructions, this lifts effective sub-10b llm throughput another 2–3x. The small model already wins on bandwidth; quantization and caching compound the win.

Tradeoffs: where small models fall over

Honesty required: sub-10B models degrade on:

• Multi-step reasoning with ambiguous constraints
• Long-context synthesis (>32k tokens)
• Tool-use orchestration requiring precise schema adherence
• Low-resource languages or domain-heavy jargon

I have seen a 7B model confidently emit malformed function calls in an agent loop, burning more tokens in retries than a single 70B call would have cost. If the task is user-facing and high-stakes (legal, medical triage), escalate.

Measure quality with offline evals before deploying. A minimal rubric:

def eval_small_vs_large(dataset, small_fn, large_fn):
    small_fail = sum(1 for x in dataset if not small_fn(x)["ok"])
    large_fail = sum(1 for x in dataset if not large_fn(x)["ok"])
    return small_fail / len(dataset), large_fail / len(dataset)

If small_fail is within 2% of large_fail on your real distribution, ship small.

Architecture patterns: routing by task difficulty

A single model endpoint is a mistake. Route by task: cheap classifier first, escalation path for low-confidence outputs. At the gateway layer, send a routing directive:

{
  "model": "auto",
  "messages": [{"role": "user", "content": "Explain quantum entanglement"}],
  "routing": {
    "prefer": "sub-10b",
    "fallback_to": "70b",
    "if_confidence_below": 0.8
  }
}

An OpenAI-compatible gateway such as n4n.ai honors client routing directives and forwards provider cache-control hints, so you can pin bulk traffic to a 7B while retaining automatic fallback when a provider is degraded. This keeps sub-10b llm throughput high without sacrificing coverage.

Cost per token is not the whole story

Per-token pricing favors small models, but the real saving is concurrency. A 7B on an L4 processes 5k input + 200 output tokens in ~0.3 s. At 10 concurrent streams you get 3k tok/s sustained. A 70B needs two A100s for similar latency, costing 8x more per hour while delivering lower QPS. Throughput per dollar is the metric that matters; sub-10b llm throughput dominates that ratio.

Decision framework

Use this filter when scoping a new LLM feature:

• Is the task narrow and verifiable? → sub-10B
• Can you tolerate <95% accuracy with a retry/escalation path? → sub-10B
• Does it need >32k context or chain-of-thought over many steps? → larger
• Is it revenue-critical with zero tolerance for hallucination? → larger + RAG

If uncertain, A/B a 7B behind a feature flag and watch the eval delta.

Takeaway

Default to sub-10B. The sub-10b llm throughput multiplier is too large to ignore for the bulk of enterprise inference. Build a routing layer that escalates only on measured failure, and you cut infrastructure spend while improving latency. Ship small, measure relentlessly, scale model size as a last resort.