How to cut LLM costs with automatic model routing

Most teams overspend on LLM inference because they hardcode the most capable model into every code path. Automatic model routing for cost savings flips this: you send each request to the cheapest model that can satisfy the task, with fallback to stronger models only when needed. This guide shows you how to implement that routing end to end with an OpenAI-compatible gateway.

What you need

• Python 3.10+ and the openai SDK (pip install openai).
• An API key for a gateway that exposes an OpenAI-compatible /v1/chat/completions endpoint and honors routing directives.
• A written list of your application’s task types and the minimum quality bar for each.

If you skip the task inventory, you will either route too aggressively and break outputs, or route too conservatively and save nothing.

Step 1: Classify workloads by complexity tier

Break your prompts into three or four buckets. A practical split:

• Trivial: classification, PII redaction, format conversion.
• Light: extraction, short summarization, templated rewrites.
• Standard: multi-paragraph summarization, moderate reasoning.
• Hard: agent planning, complex codegen, ambiguous instruction following.

Encode this as data, not scattered if statements:

# routing_tiers.py
TIERS = {
    "trivial": ["classify", "redact", "to_json"],
    "light": ["extract", "summarize_short", "rephrase"],
    "standard": ["summarize_long", "draft"],
    "hard": ["plan", "codegen", "analyze"],
}

MODEL_HINTS = {
    "trivial": "cheap",
    "light": "cheap",
    "standard": "mid",
    "hard": "powerful",
}

The hints are arbitrary strings your gateway maps to actual models. The point is that automatic model routing for cost savings starts with explicit, auditable tier definitions.

Step 2: Point your client at a routing gateway

A gateway such as n4n.ai provides one OpenAI-compatible endpoint covering 240+ models and automatic fallback when a provider is rate-limited or degraded, which removes the need to wire multiple provider SDKs. You keep the OpenAI client and swap the base_url.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_API_KEY",
)

The gateway decides the concrete model from your routing hint and falls back transparently if the cheap provider returns 429s. Your application code stays identical across providers.

Step 3: Send routing directives per request

Pass the tier as a routing directive in the request. The gateway honors client routing directives via headers or extra body fields; use whichever your client supports. With the OpenAI SDK:

def complete(task: str, prompt: str):
    tier = next(t for t, tasks in TIERS.items() if task in tasks)
    hint = MODEL_HINTS[tier]
    return client.chat.completions.create(
        model="gpt-4o-mini",  # fallback model; gateway overrides via hint
        messages=[{"role": "user", "content": prompt}],
        extra_headers={"X-Route-Tier": hint},
    )

resp = complete("classify", "Is this spam? 'Win a free iPhone now'")
print(resp.choices[0].message.content)

If your gateway uses a body field instead, pass extra_body={"route_tier": hint}. The key engineering discipline: never call model= with a fixed powerful model from app code. Let the directive drive selection.

Automatic model routing for cost savings only works if every entry point in your codebase funnels through this complete() wrapper. Audit your repos for direct SDK calls and replace them.

Step 4: Forward provider cache-control hints

Re-sending the same long system prompt or knowledge base context wastes tokens. Gateways forward provider cache-control hints, so mark static prefixes as cacheable.

For Anthropic-backed routes behind the gateway:

{
  "model": "claude-3-haiku",
  "messages": [
    {
      "role": "user",
      "content": "System context: ...long legal doc...",
      "cache_control": {"type": "ephemeral"}
    },
    {"role": "user", "content": "What is the termination clause?"}
  ]
}

In Python:

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helper."},
        {"role": "user", "content": BIG_CONTEXT, "cache_control": {"type": "ephemeral"}},
        {"role": "user", "content": user_question},
    ],
    extra_headers={"X-Route-Tier": "standard"},
)

Cache hits show up as reduced prompt_tokens on repeated calls. This compounds the savings from routing because the expensive part (the context) is paid once.

Step 5: Track per-token metering to confirm savings

The gateway returns standard OpenAI usage objects. Log them with the route tier:

resp = complete("extract", "Pull name and date from: John met on 2024-01-02")
usage = resp.usage.model_dump()
print({
    "task": "extract",
    "tier": "cheap",
    "prompt_tokens": usage["prompt_tokens"],
    "completion_tokens": usage["completion_tokens"],
    "total_tokens": usage["total_tokens"],
})

Per-token usage metering lets you compute effective cost per task type. Compare this week’s cheap tier average against what the same traffic would have cost on your previous fixed gpt-4o baseline. The delta is your automatic model routing for cost savings realized, not theorized.

Step 6: Verify end-to-end with a test script

Write a small harness that sends one request per tier and asserts the response is non-empty and the usage object is present.

curl -s https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Route-Tier: cheap" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Classify: happy"}]}' \
  | python -m json.tool

Expected output includes choices[0].message.content and usage.total_tokens > 0. If the X-Route-Tier header is ignored, the gateway will still answer using the fallback model—so check your gateway’s dashboard or response headers for the resolved model ID.

A stronger verification: run a labeled eval set through both a fixed powerful model and your routed wrapper. Measure accuracy and token cost. If accuracy drops more than your quality bar allows, promote that task to a higher tier.

Gotchas

Latency asymmetry. Cheap models are faster, but fallback adds a retry hop. Set a timeout on the client and treat gateway timeout as a hard error, not a silent downgrade.

Quality regression. Routing summarize_long to a 8B model can silently drop section coverage. Keep a golden set and run it in CI.

Fallback storms. If the cheap provider is consistently degraded, the gateway will failover to the powerful model for 100% of traffic. You just traded cost for reliability—monitor fallback rate.

Cache hint mismatches. Not every provider supports the same cache-control schema. The gateway forwards hints; if the downstream provider rejects them, you may get a 400. Test each tier’s cache payload against the resolved model.

Automatic model routing for cost savings is not a one-time toggle. It is a feedback loop: define tiers, route, meter, eval, adjust. Do that and your inference bill drops while your outputs stay within spec.