When to route requests to cheaper models automatically

Most LLM applications spend the majority of their inference budget on requests that a smaller model could handle. Automatic routing to cheaper models reduces cost without noticeable quality loss, but only when you route based on measurable signal rather than naive round-robin. This guide lays out an ordered path from baseline measurement to a production router with fallback.

1. Define your cost and quality baseline

You cannot optimize what you do not measure. Pull a week of production traffic and split spend by model, token volume, and endpoint latency. If you already call an OpenAI-compatible gateway, the response usage field gives per-token counts; some gateways also expose per-token usage metering at the account level.

Compute relative cost rather than absolute dollars. Model prices shift, but the ratio between a frontier model and a small instruction-tuned model stays roughly an order of magnitude. Use placeholder constants so the logic survives pricing changes:

PRICE_PER_1K = {
    "frontier-large": 0.01,   # placeholder, replace with live values
    "small-8b": 0.001,
    "mini-1b": 0.0002,
}

def est_cost(model: str, prompt_tok: int, completion_tok: int) -> float:
    return (prompt_tok + completion_tok) / 1000 * PRICE_PER_1K[model]

Establish a quality bar with a small eval set. Even 50 hand-labeled examples per task type will reveal whether a 7B model drops critical fields in extraction or fails to follow a strict tone. Store these as regression tests before you route any traffic.

2. Classify requests by expected difficulty

Routing needs a feature vector. Start with cheap, observable signals that you can compute in under a millisecond:

• User-facing tier (free vs paid)
• Prompt token count (approx len(text)//4)
• Endpoint or function name (summarize, classify, agent_plan)
• Structured-output requirement (JSON schema, yes/no)
• Historical failure rate for the route

A simple policy function beats a fancy classifier early on:

from dataclasses import dataclass

@dataclass
class Req:
    task: str
    prompt_tokens: int
    user_tier: str
    needs_json: bool

def route_key(r: Req) -> str:
    if r.task in ("embed", "classify"):
        return "mini"
    if r.needs_json and r.task != "extract_simple":
        return "large"
    if r.task == "summarize" and r.prompt_tokens < 2000:
        return "small"
    if r.user_tier == "free" and r.prompt_tokens < 4000:
        return "small"
    return "large"

This is a policy, not ML. You can later train a logistic regression on logged features, but the rule-based version gives you a safe baseline and a clear audit trail.

3. Implement the routing layer

Keep the router a pure function that returns a model identifier. The caller passes that to the client. Using an OpenAI-compatible interface means you only change the model string, which makes incremental rollout trivial.

MODEL_MAP = {
    "mini": "provider/mini-1b",
    "small": "provider/small-8b",
    "large": "provider/frontier-large",
}

def complete(r: Req, prompt: str):
    key = route_key(r)
    model = MODEL_MAP[key]
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )

If you need to force a specific provider path, send a routing directive in the request body when your gateway supports it:

{
  "model": "auto",
  "messages": [{"role": "user", "content": "..."}],
  "routing": {
    "prefer": ["provider/small-8b"],
    "fallback": ["provider/frontier-large"]
  }
}

In a serverless edge context, the same logic fits a few lines of TypeScript:

export function pickModel(r: {task: string, tokens: number}): string {
  if (r.task === "classify") return "provider/mini-1b";
  if (r.task === "summarize" && r.tokens < 2000) return "provider/small-8b";
  return "provider/frontier-large";
}

4. Handle fallback and degradation

Small models fail silently: they return malformed JSON, drop constraints, or hallucinate structure. Your router must catch that and retry on a larger model. Providers also rate-limit and degrade. An inference gateway such as n4n.ai provides automatic fallback when a provider is rate-limited or degraded, and honors client routing directives—so you can keep the retry logic minimal and focus on semantic validation.

class RouteError(Exception): pass

def safe_complete(r: Req, prompt: str):
    try:
        resp = complete(r, prompt)
        content = resp.choices[0].message.content
        if r.needs_json and not is_valid_json(content):
            raise RouteError("json invalid")
        return resp
    except (RouteError, RateLimitError, TimeoutError):
        # single upgrade, no cascade
        return client.chat.completions.create(
            model=MODEL_MAP["large"],
            messages=[{"role": "user", "content": prompt}]
        )

Tradeoff: fallback adds tail latency. Cap upgrades to one hop and set a tight timeout on the small model so the large call still meets SLA.

5. Meter and close the loop

Per-token metering lets you attribute savings and detect drift. Log the chosen route, the actual model used after fallback, and token counts on every call.

log = {
    "route_key": key,
    "model_used": resp.model,
    "prompt_tokens": resp.usage.prompt_tokens,
    "completion_tokens": resp.usage.completion_tokens,
    "fallback": resp.model != MODEL_MAP[key],
}

Aggregate daily by route_key. If the small bucket shows a double-digit fallback rate to large, either tighten the classifier or fine-tune the small model on those specific failures. Automatic routing to cheaper models only pays off when the fallback rate stays low for the bulk of traffic.

Set alerts on two metrics:

• Fallback rate per route > 10%
• Quality score on sampled small outputs below baseline

6. Common pitfalls and tradeoffs

Prompt drift

Prompts tuned for a frontier model often fail on smaller ones. Maintain separate prompt templates per model family, or use a middleware that rewrites instructions. Do not assume a single prompt works across a 1B and a 70B model.

Cache fragmentation

Provider cache-control hints are model-specific. Automatic routing to cheaper models can split traffic and reduce cache hit rate, raising cost. Forward cache-control hints only when the route is stable, and avoid routing the same prefix to three different model sizes.

Latency vs cost

A router that calls a classifier model defeats the purpose. Keep routing logic local, rule-based, and sub-millisecond. If you need ML scoring, run it offline on logged traffic to generate rules, not per request.

Hidden quality debt

Cheap routing hides regressions. Sample 1% of small outputs for human review; alert on schema violations. A 2% silent failure rate on extracted invoices is worse than a 20% cost increase.

Provider heterogeneity

Different providers name similar-sized models differently and have varying tool-calling support. Abstract model names behind your MODEL_MAP and test tool calls explicitly per route.

7. Ordered deployment path

1. Instrument current traffic with per-token metering and tag each request with task type.
2. Build a static route map keyed on task, prompt size, and user tier using the rule function above.
3. Shadow the router: run it mirrored to the large model, compare outputs on your eval set.
4. Enable automatic fallback on parse or validation error, capped at one upgrade.
5. Review weekly fallback rates and quality samples; promote stable routes to default.
6. Optimize only after rules plateau—introduce lightweight ML scoring on logged features, not live calls.

Automatic routing to cheaper models is not a one-time switch. It is a control loop: classify, route, measure, tighten. Done right, it cuts spend by sending the boring 80% to models that are good enough, while reserving the expensive frontier for the tasks that genuinely need it.