Cost-based LLM routing is a strategy that selects the least expensive model capable of satisfying a request’s quality requirements. Instead of defaulting to a single premium model for every call, the router evaluates each request against a ranked list of models — ordered by price per token — and chooses the first one that meets the task’s constraints. This approach can reduce inference spend by 40–70% compared to single-model deployments without measurable quality regression for most workloads.
How cost-based routing works
The routing decision happens at request time. When a request arrives, the router inspects metadata — task type, required context length, latency budget, quality threshold — and matches it against a capability matrix. Each model in the matrix carries a price (input tokens, output tokens, sometimes cached tokens) and a capability profile (max context, supported modalities, benchmark scores on relevant evals). The router picks the cheapest model whose profile clears the request’s minimum bar.
# Simplified routing logic
MODEL_CATALOG = [
{"name": "gpt-4o-mini", "input_price": 0.15, "output_price": 0.60, "max_ctx": 128_000, "quality_tier": 1},
{"name": "claude-3-haiku", "input_price": 0.25, "output_price": 1.25, "max_ctx": 200_000, "quality_tier": 1},
{"name": "gpt-4o", "input_price": 2.50, "output_price": 10.00, "max_ctx": 128_000, "quality_tier": 2},
{"name": "claude-3-5-sonnet", "input_price": 3.00, "output_price": 15.00, "max_ctx": 200_000, "quality_tier": 3},
]
def route_request(task_type: str, min_quality_tier: int, est_input_tokens: int, est_output_tokens: int, max_latency_ms: int):
candidates = [m for m in MODEL_CATALOG if m["quality_tier"] >= min_quality_tier]
# Sort by estimated total cost
candidates.sort(key=lambda m: m["input_price"] * est_input_tokens + m["output_price"] * est_output_tokens)
return candidates[0]["name"] if candidates else NoneThe capability matrix is the critical artifact. It must be maintained continuously as providers update pricing, release new models, or deprecate old ones. A stale matrix routes requests to models that no longer exist or misprices them relative to current rates. Most teams automate this by ingesting provider pricing APIs and running nightly eval suites to keep quality tiers calibrated.
Why it matters for production systems
LLM spend scales linearly with token volume. A single-model strategy forces you to pay premium rates for every request — classification, extraction, summarization, chat — even when a $0.15/M model handles the task indistinguishably from a $10/M model. At millions of requests per month, that delta is the difference between a manageable line item and a budget crisis.
Cost-based routing also creates leverage in vendor negotiations. When your routing layer can shift 80% of traffic to a cheaper provider within minutes, you’re not locked into any single provider’s pricing tier. This dynamic is why gateways like n4n.ai expose routing directives in the request headers — the client retains control while the gateway executes the decision across 240+ models.
Beyond direct savings, the pattern forces discipline on your evaluation pipeline. You cannot route safely without knowing which models actually satisfy which tasks. That knowledge requires systematic evals, which most teams skip when they only ever test one model.
Concrete example: a document QA pipeline
Consider a RAG system answering questions over internal documentation. The pipeline has three stages: query rewriting, retrieval, and answer generation. Each stage has different quality requirements.
{
"stages": [
{
"name": "query_rewrite",
"task": "Rewrite user query for better retrieval",
"min_quality_tier": 1,
"est_tokens": {"input": 200, "output": 50},
"max_latency_ms": 500
},
{
"name": "answer_generation",
"task": "Generate final answer with citations",
"min_quality_tier": 2,
"est_tokens": {"input": 4000, "output": 800},
"max_latency_ms": 3000
}
]
}Query rewriting is a constrained transformation — it needs instruction following but not deep reasoning. Tier 1 models (GPT-4o-mini, Claude 3 Haiku, Gemini 1.5 Flash) handle this reliably at ~$0.20/M blended. Answer generation needs synthesis across retrieved chunks and citation discipline; tier 2 (GPT-4o, Claude 3.5 Sonnet) is the floor. Routing each stage independently cuts the blended cost per query from ~$0.045 (all Sonnet) to ~$0.012 (mixed), a 73% reduction.
The routing layer sits in front of the model calls. Each stage passes its routing constraints; the gateway resolves them to a concrete model ID. If the primary model for a tier is rate-limited or degraded, the gateway falls back to the next cheapest model in the same tier — preserving the cost target while maintaining availability.
# Gateway-side fallback within a quality tier
TIER_MODELS = {
1: ["gpt-4o-mini", "claude-3-haiku", "gemini-1.5-flash"],
2: ["gpt-4o", "claude-3-5-sonnet"],
3: ["claude-3-opus", "gpt-4-turbo"],
}
def select_model(tier: int, excluded: set[str] = None) -> str:
excluded = excluded or set()
for model in TIER_MODELS[tier]:
if model not in excluded and is_healthy(model):
return model
# Escalate to next tier if entire tier is down
return select_model(tier + 1, excluded)Common misconceptions
“Cheapest model means worst quality”
Price correlates with capability, but not perfectly. Distilled and optimized smaller models (GPT-4o-mini, Claude 3 Haiku, Gemini Flash) often match their larger predecessors on narrow tasks like classification, entity extraction, and structured output generation. The quality gap appears in open-ended reasoning, long-context synthesis, and creative writing. Cost-based routing exploits this: it uses cheap models where they’re strong and reserves expensive models for where they’re necessary.
“Routing adds unacceptable latency”
A well-implemented routing decision takes microseconds — it’s a dictionary lookup and a sort over ~10 items. The network hop to the gateway adds 1–5ms in the same region. Compare that to the 500–3000ms latency of the model call itself. The routing overhead is negligible. What does add latency is routing to a model in a different region or a provider with cold-start penalties. Keep your model catalog region-aware.
“You need perfect evals before starting”
You need some evals, but they don’t need to be perfect. Start with a binary pass/fail on a representative sample per task type. Run the candidate models. Mark the cheapest passer as tier 1 for that task. Iterate. The eval suite improves over time; the routing layer just needs a signal to act on. Waiting for comprehensive benchmarks before enabling routing means leaving money on the table indefinitely.
“Cost-based routing ignores latency and reliability”
It doesn’t have to. The routing constraints are multidimensional: cost, latency, reliability, context window, modality support. The router optimizes cost subject to the other constraints. If a request has a 500ms latency budget, the router excludes models whose p95 exceeds that — even if they’re cheaper. If a provider is returning 5xx errors, the gateway’s health checks exclude it from the candidate pool. Cost is the objective function; the constraints define the feasible set.
“One routing policy fits all requests”
Different request classes need different policies. A customer-facing chatbot needs higher quality tiers than an internal classification job. A real-time feature needs stricter latency budgets than a batch job. The routing layer should accept policy overrides per request — via headers, user tier, or feature flags — so the same infrastructure serves diverse workloads without hardcoding exceptions.
Building the capability matrix
The matrix is a living document. Here’s a minimal schema:
{
"models": [
{
"id": "gpt-4o-mini",
"provider": "openai",
"pricing": {"input": 0.15, "output": 0.60, "cached_input": 0.075},
"limits": {"max_context": 128000, "max_output": 16384},
"capabilities": {"vision": true, "json_mode": true, "parallel_tools": true},
"benchmarks": {
"mmlu": 82,
"gpqa": 40,
"humaneval": 87,
"internal_qa_f1": 0.89,
"internal_extraction_f1": 0.94
},
"latency_p50_ms": 450,
"latency_p95_ms": 1200,
"region_availability": ["us-east-1", "eu-west-1"],
"quality_tiers": {"general": 1, "coding": 2, "reasoning": 2}
}
]
}Key fields:
- Pricing: Per million tokens. Include cached input pricing where providers offer it (OpenAI, Anthropic, Google all do now).
- Internal benchmarks: Your eval scores matter more than public leaderboards. Run your task-specific evals on every model you might route to.
- Quality tiers per task type: A model might be tier 1 for extraction but tier 2 for coding. Encode this granularity.
- Latency percentiles: p50 for typical experience, p95 for SLA compliance.
- Region availability: Critical for latency-constrained routing.
Update this matrix weekly at minimum. Automate pricing ingestion from provider APIs. Run evals nightly on new model versions. Deprecate models when providers announce sunset dates.
Request-level routing directives
Clients should express intent, not implementation. The request carries what the caller needs; the gateway decides how to satisfy it.
POST /v1/chat/completions
Authorization: Bearer sk-...
Content-Type: application/json
X-Routing-Policy: cost-optimized
X-Min-Quality-Tier: 2
X-Max-Latency-Ms: 2000
X-Required-Capabilities: ["json_mode", "vision"]
{
"messages": [...],
"response_format": {"type": "json_object"}
}The gateway validates that the requested capabilities exist in at least one model meeting the quality tier and latency budget. If not, it returns a 400 with a clear error — “no model satisfies vision + tier 2 + 2000ms in region us-east-1” — rather than silently degrading quality.
This design keeps routing logic centralized. Clients don’t hardcode model names. When a new cheaper model clears the quality bar, the gateway starts using it automatically. Clients only update their constraints when their requirements change.
Measuring the impact
Track three metrics to verify routing works:
- Blend cost per 1k tokens: Weighted average across all routed requests. Should trend down as you add cheaper models to tiers.
- Tier distribution: Percentage of requests served by each quality tier. A healthy system routes 60–80% to tier 1, 15–30% to tier 2, <5% to tier 3.
- Quality regression rate: Automated eval pass rate on production traffic samples, segmented by tier. Tier 1 should pass at ≥95% of the tier 3 baseline for its assigned tasks.
-- Daily cost blend
SELECT
DATE(created_at) as day,
SUM(estimated_cost_usd) / SUM(total_tokens) * 1000 as cost_per_1k_tokens,
COUNT(*) FILTER (WHERE quality_tier = 1) * 100.0 / COUNT(*) as pct_tier_1,
COUNT(*) FILTER (WHERE quality_tier = 2) * 100.0 / COUNT(*) as pct_tier_2,
COUNT(*) FILTER (WHERE quality_tier = 3) * 100.0 / COUNT(*) as pct_tier_3
FROM request_logs
GROUP BY day
ORDER BY day DESC;Alert when tier 1 share drops below 50% (routing may be misconfigured) or when quality regression exceeds 2% week-over-week (eval thresholds may have drifted).
When not to use cost-based routing
- Single-model simplicity: If you have one workload, one model, and predictable volume, a static model choice is fine. Routing adds operational surface area.
- Hard real-time constraints: If you need guaranteed <100ms p99 and only one model meets it, routing can’t help — but you also don’t have a cost optimization problem.
- Regulatory model pinning: Some industries require auditable, fixed model versions. Routing within a pinned version set is possible but the value shrinks.
- Insufficient eval coverage: If you cannot reliably evaluate model quality on your tasks, you cannot safely route. Build evals first.
Summary
Cost-based LLM routing treats model selection as an optimization problem: minimize cost subject to quality, latency, and capability constraints. The mechanism is straightforward — a capability matrix, a request-time decision function, and fallback logic — but the prerequisite is rigorous, task-specific evaluation. Without evals, you’re guessing. With them, routing typically cuts inference spend by half or more while maintaining quality. The gateway handles the execution; your job is defining the constraints and maintaining the evidence that they’re met.