Using cheaper models for easy tasks, GPT-4 for hard ones

A task-based model routing strategy lets you send trivial classification calls to a cheap small model while reserving GPT-4 for genuine reasoning work. The pattern cuts spend without degrading output quality, but only if you define task difficulty precisely and build a classification layer that fails safe.

Why route by task difficulty

Most production LLM traffic is boring. Summarizing a ticket, extracting a name, classifying sentiment, rewriting a sentence. None of those need 100B+ parameters. Running them on GPT-4 wastes money and adds tail latency.

A task-based model routing strategy matches model capacity to cognitive load. You keep the expensive frontier model for ambiguous, multi-step, or high-stakes prompts. Everything else drops to a 7B–13B instruct model or a purpose-built classifier. The win is not just cost; it is also throughput. Small models batch better and return faster on commodity GPUs.

Define “easy” and “hard” for your workload

Generic labels fail. For a support bot, “easy” might be FAQ retrieval with a known answer template. “Hard” is policy exceptions requiring reading three documents. Write the definitions down before writing code.

• Easy: single intent, short context (<2k tokens), deterministic output format, no external tools.
• Medium: light reasoning, may need one tool call, moderate context.
• Hard: multi-document synthesis, code generation with constraints, ambiguous user goal, high business risk.

Tag each prompt template in your codebase. If you use a prompt registry, add a tier field.

{
  "prompt_id": "ticket_sentiment",
  "tier": "easy",
  "model_preference": ["mistral-7b-instruct", "gpt-3.5-turbo"]
}

Do not tier by model name. Tier by task shape. The model map can change later without touching your definitions.

Build a classification layer

Heuristics first

Before calling an LLM to classify, use cheap signals. Endpoint path, payload size, presence of keywords.

def heuristic_tier(payload: dict) -> str:
    text = payload.get("text", "")
    if len(text) < 200 and payload.get("type") == "sentiment":
        return "easy"
    if "analyze_contract" in payload.get("route", ""):
        return "hard"
    return "unknown"

Heuristics cover 80% of traffic at zero latency cost. They also never hallucinate.

LLM-based classifier for the long tail

For unknown, call a small model with a strict JSON schema. Keep the prompt tight.

CLASSIFY_PROMPT = """Classify the user request as easy, medium, or hard.
Easy: single-step, factual, short.
Hard: needs synthesis, code, or tool use.
Respond only with JSON: {"tier": "easy|medium|hard"}"""

def llm_classify(text: str) -> str:
    resp = client.chat.completions.create(
        model="mistral-7b-instruct",
        messages=[{"role": "system", "content": CLASSIFY_PROMPT},
                  {"role": "user", "content": text}],
        temperature=0.0,
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)["tier"]

The classifier itself runs on the cheap tier. If it errors, default to hard. Never let a classifier failure silently downgrade a task.

Implement the routing call

Wire the tier to a model map. Use an OpenAI-compatible client so swapping providers is config-only.

MODEL_MAP = {
    "easy": "mistral-7b-instruct",
    "medium": "gpt-3.5-turbo",
    "hard": "gpt-4o",
}

# Point the client at a single OpenAI-compatible endpoint (n4n.ai exposes one covering 240+ models)
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["LLM_KEY"])

def complete(tier: str, messages: list):
    model = MODEL_MAP.get(tier, "gpt-4o")
    return client.chat.completions.create(model=model, messages=messages)

The gateway handles provider rate limits and degradations, so you don’t need retry logic for every vendor. Your code only picks a model ID.

Escalate on failure

Cheap models hallucinate or return malformed JSON. Validate output. If validation fails, re-call with the next tier up.

def safe_complete(tier: str, messages: list, validator: callable):
    resp = complete(tier, messages)
    content = resp.choices[0].message.content
    if not validator(content):
        # escalate
        return complete("hard", messages)
    return resp

This fail-safe approach protects quality. The extra cost only hits when the small model would have produced a bad answer anyway.

Pitfalls and tradeoffs

Added latency from classification

Running a classifier adds a round trip. For easy tasks under 50ms, a 20ms classifier is fine. For synchronous user-facing calls, cache the tier per prompt template so you classify once at deploy time, not per request.

Misclassification is asymmetric

Routing a hard task to a small model can cause a wrong refund approval. Routing an easy task to GPT-4 just costs cents. Bias toward the expensive model when confidence is low. Set your heuristic to return hard on any missing field.

Cache behavior

Provider caching keys on model ID and prompt prefix. If you route the same prompt to different models, you lose cache hits. Forward provider cache-control hints unchanged. Design your routing so stable prefixes stay consistent per tier; do not inject tier-specific instructions that break the cache key.

Temperature and sampling differences

Small models often need lower temperature to stay formatted. GPT-4 can handle higher creativity. Encode sampling params in the tier map, not the call site.

SAMPLING_MAP = {
    "easy": {"temperature": 0.0, "max_tokens": 64},
    "hard": {"temperature": 0.7, "max_tokens": 1024},
}

Token metering

You need per-token attribution to know if the strategy works. Log usage from each response, tagged by tier and prompt_id. Without metering, you are guessing. A simple structlog line per call is enough to start.

import structlog
log = structlog.get_logger()

def complete(tier: str, messages: list):
    model = MODEL_MAP.get(tier, "gpt-4o")
    resp = client.chat.completions.create(model=model, messages=messages)
    log.info("llm.call", tier=tier, model=model,
             prompt_tokens=resp.usage.prompt_tokens,
             completion_tokens=resp.usage.completion_tokens)
    return resp

Operational checklist

1. Inventory your top 20 prompt templates by volume.
2. Label each with a tier based on defined criteria.
3. Implement heuristic classification in the request path.
4. Add an LLM classifier only for ambiguous templates.
5. Route via a single OpenAI-compatible client with fallback.
6. Validate outputs; escalate to GPT-4 on failure.
7. Meter tokens per tier; review weekly.

A task-based model routing strategy is not a one-time setup. Model catalogs change monthly. Re-evaluate tiers when a new small model beats your current medium tier.

When a gateway earns its keep

If you run multi-region with several providers, the routing logic should pick a model ID, not a vendor. A gateway that honors client routing directives and provides per-token usage metering removes the fallback boilerplate. You focus on tier definitions; the endpoint deals with which provider is healthy. The alternative is writing provider-specific retry and auth code for every model you touch.

That is the whole game: define difficulty, classify cheaply, route, escalate, measure. The teams that ship this early stop worrying about LLM bills and start shipping features.