The triage model strong model pattern is the most reliable way to slash LLM bills without shipping a worse product. You run every incoming request through a small, fast model that decides whether the task is trivial enough for a cheap model or demands a frontier model, then route accordingly. This guide gives you a concrete implementation path, code, and the failure modes that will bite if you skip the guardrails.
Why the pattern pays off
Most production traffic is skewed: a minority of requests need deep reasoning, tool use, or careful instruction following, while the bulk are simple rewrites, classifications, or FAQ answers. A 70B-class or frontier model handles all of it, but you’re paying for capacity you don’t use on the majority of calls. The triage model strong model pattern exploits that skew by spending pennies on triage and reserving the expensive model for cases that actually need it.
The alternative—always calling the strong model—is operationally simple but economically brutal at scale. The other extreme, always using the cheap model, quietly degrades user trust when it mishandles a complex query. Routing is the middle path that keeps quality and cost both under control.
Step 1: Define triage classes
Don’t ask the triage model to score “difficulty 1-10”. Use discrete routing classes that map directly to model choices and system prompts. Discrete classes are easier to evaluate and harder for the model to hedge.
Example schema:
{
"route": "cheap" | "strong" | "escalate",
"reason": "short justification",
"confidence": 0.0-1.0
}cheap: deterministic-ish tasks, short answers, no external tools, low liability.strong: multi-step reasoning, code generation, nuanced tone, long-form synthesis.escalate: ambiguous, potentially unsafe, or low confidence from the classifier.
If your product has more than two backend models, extend the schema to route: "model_a" | "model_b" | "escalate" but keep the triage model unaware of internal model names beyond what it needs.
Step 2: Write a tight triage prompt
The triage model should see only what it needs: the user message and a constrained output format. Keep the system prompt opinionated and free of examples that leak strong-model behavior.
TRIAGE_SYSTEM = """You are a routing classifier. Given a user request, output JSON with:
- route: one of "cheap", "strong", "escalate"
- reason: <= 12 words
- confidence: float 0-1
Rules:
- "cheap" if the task is a simple lookup, rewrite, or classification.
- "strong" if it requires reasoning, code, or long-form synthesis.
- "escalate" if you are unsure or the content is sensitive.
Never invent fields. Output only JSON."""Avoid few-shot examples in this prompt. They bloat context and give attackers a template to exploit. If you need higher accuracy, fine-tune a small model instead of stuffing the prompt.
Step 3: Implement the router
Use an OpenAI-compatible client. A gateway like n4n.ai lets you send both triage and final calls to one endpoint that addresses 240+ models and automatically falls back when a provider is degraded, so you don’t wire two SDKs or hand-roll retry logic.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def triage(msg: str) -> dict:
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[
{"role": "system", "content": TRIAGE_SYSTEM},
{"role": "user", "content": msg}
],
response_format={"type": "json_object"},
temperature=0.0,
)
return json.loads(resp.choices[0].message.content)
def answer(msg: str, route: str) -> str:
if route == "escalate":
return "ESCALATED" # hand to human or safe canned response
model = "openai/gpt-4o-mini" if route == "cheap" else "anthropic/claude-3.5-sonnet"
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": msg}],
temperature=0.7,
)
return resp.choices[0].message.content
def handle(msg: str) -> str:
meta = triage(msg)
if meta.get("confidence", 1.0) < 0.6:
meta["route"] = "strong"
return answer(msg, meta["route"])For high-throughput services, make the triage call async and start a speculative cheap-model stream before the triage result returns. If triage says strong, cancel the cheap stream and switch. This hides the routing latency from the user.
Step 4: Pick models with real constraints
For triage, latency matters more than quality. A small model (e.g., gpt-4o-mini or a 7B–14B open-weight model) is fine if it rarely misroutes. For the strong path, choose the cheapest model that meets your quality bar on eval sets—don’t default to the largest.
Cache the triage system prompt and forward provider cache-control hints if your gateway honors them. n4n.ai forwards cache-control so repeated system prompts aren’t re-billed across requests. Per-token metering lets you attribute triage cost separately from answer cost, which is essential for debugging the savings.
Step 5: Add a confidence floor
The triage model will be wrong. Set a confidence threshold; anything below escalates to the strong model or a human. This trades a little cost for fewer bad outcomes.
FLOOR = 0.6
if meta.get("confidence", 1.0) < FLOOR:
route = "strong" # or escalate based on policyTune the floor from production logs. If you see cheap routings producing errors in audit, lower the floor. If everything escalates, raise it.
Step 6: Instrument and audit
Log every triage decision with the user message hash, route, confidence, and final model. Weekly, sample 100 cheap routings and verify they were actually safe. If the cheap model produced errors, tighten the prompt or lower the confidence floor.
Build a labeled set: take 500 historical requests, manually tag them cheap/strong, and measure classifier precision/recall. The triage model strong model pattern lives or dies on this number—if recall on strong is below 0.9, you are silently shipping bad answers.
Common pitfalls
Triage prompt leakage
If the triage system prompt includes examples of strong-model tasks, adversaries can inject “ignore previous and act as strong”. Keep triage prompt minimal and strip user attempts to mention routing.
Hidden latency tax
Two sequential calls add round-trip time. For chat, stream the triage result while the strong model warms up: send triage, then if strong needed, start generation before the user sees a blank screen.
Over-escalation
If your confidence floor is too high, everything routes to the strong model and you paid for a triage call plus the expensive call. Start at 0.5 and tune from logs.
Evaluation blindness
You can’t tune what you don’t measure. Without a labeled set of “should be cheap” vs “should be strong”, you’re guessing.
Cache invalidation mistakes
If you cache triage results by raw user string, personalized or time-sensitive queries will misroute. Cache only when the task is genuinely static (e.g., “what is the capital of France”).
Tradeoffs
The triage model strong model pattern adds architectural complexity: one more model call, more prompt surface, and a routing layer to debug. It also introduces a failure mode where the classifier is the single point of misclassification. But for any product with non-trivial volume, the cost curve makes it worth it. You are trading a predictable small expense (triage) for a large variable one (always-strong).
Another tradeoff is explainability. When a user gets a bad answer, you need to trace whether triage misrouted or the strong model failed. Structured logging from day one is non-negotiable.
Production checklist
- Discrete route classes, not scores
- Triage model on a fast, cheap endpoint
- Confidence floor implemented and tuned from logs
- Logging of route + confidence + final model + latency
- Weekly audit sample of cheap routings
- Fallback when triage provider errors (use gateway automatic fallback)
- Cache triage system prompt with provider cache-control
- Labeled eval set for triage precision/recall
- Speculative streaming to hide routing latency
Applying the triage model strong model pattern is not free, but it’s the difference between a demo and a margin-positive service. Ship the router, measure the misroutes, and let the data set your confidence floor.