Most multimodal apps default to the most expensive vision model because it’s the safest bet for accuracy. To route vision requests to cheapest model that still meets your quality bar, you need explicit capability checks and a deterministic selector in front of your inference gateway. This guide walks through building that router as code, not a black box, and shows how to verify it in production.
Step 1: Build a capability matrix for vision models
Before you can route vision requests to cheapest model, you need a factual inventory of which models accept images and what they support. Pull this from provider docs, not marketing. At minimum, record context window, max images per call, supported image types, and a relative cost tier based on published token prices.
{
"gpt-4o-mini": {"vision": true, "max_images": 10, "cost_tier": 2, "min_quality": "medium"},
"gemini-1.5-flash": {"vision": true, "max_images": 1, "cost_tier": 1, "min_quality": "medium"},
"claude-3-5-haiku": {"vision": true, "max_images": 20, "cost_tier": 1, "min_quality": "high"},
"llama-3.2-11b-vision": {"vision": true, "max_images": 4, "cost_tier": 1, "min_quality": "low"}
}Cost tier 1 is cheapest, 3 is premium. Do not guess exact prices; use the provider’s published per-token rates if you want absolute sorting, but tiers are enough for routing logic. Competitor gateways often expose an “auto” route that picks any model across providers, but they rarely let you constrain the decision to cheapest-only or show the selection rule. Owning the matrix means you control the rule.
Watch the fine print
Some models silently downsample images; others charge extra for image tokens. A model marked vision:true may reject PDFs or exceed context with high-res scans. Encode those constraints as booleans or integers in your matrix. If a provider adds a new cheap vision model, you add one line—no support ticket.
Step 2: Score the incoming request
Write a function that inspects the payload before sending. Count images, check dimensions, and classify task complexity. A simple heuristic: OCR on a receipt is low complexity; diagram reasoning is high. This runs in microseconds and gives your router the facts to filter models.
def request_constraints(payload):
images = payload.get("images", [])
n = len(images)
total_pixels = sum(img.get("width", 0) * img.get("height", 0) for img in images)
task = payload.get("task", "caption")
complexity = {"ocr": 1, "caption": 1, "describe": 2, "reason": 3}.get(task, 2)
return {"n_images": n, "pixels": total_pixels, "complexity": complexity}In practice, you should also inspect MIME types. A image/webp may be fine for one model and unsupported for another. Extend the constraints dict with formats if you routinely hit format rejections. The point is to fail locally before spending a network round-trip.
Step 3: Select the cheapest capable model
Now apply the filter. Keep only models where vision is true, max_images >= n_images, and a quality score meets complexity. Then sort by cost_tier. This is the core of how you route vision requests to cheapest model without sacrificing the ability to read the image.
def select_model(matrix, constraints):
candidates = []
for name, spec in matrix.items():
if not spec["vision"]:
continue
if spec["max_images"] < constraints["n_images"]:
continue
if spec["min_quality"] == "low" and constraints["complexity"] > 2:
continue
if spec["min_quality"] == "medium" and constraints["complexity"] > 3:
continue
candidates.append((spec["cost_tier"], name))
if not candidates:
raise ValueError("no capable model")
candidates.sort()
return candidates[0][1]You can extend the matrix with latency percentiles from your own telemetry. If two models share a cost tier, break ties by recent p95 latency. The selector stays pure and testable.
Step 4: Call the gateway with an explicit model directive
Most OpenAI-compatible gateways let you pin the model via the model field. If you use a gateway that honors client routing directives and forwards provider cache-control hints, set the model exactly as selected and pass cache headers to avoid re-paying for unchanged images.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
resp = client.chat.completions.create(
model=select_model(matrix, request_constraints(payload)),
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": img["url"]}}
for img in payload["images"]
] + [{"type": "text", "text": payload["prompt"]}]
}],
extra_headers={"x-cache-control": "image=300"} # hint to reuse image tokens
)The gateway forwards the directive upstream. If the model is unavailable, you handle it in the next step. Note that some gateways ignore the model field when set to a family alias; always log the returned model field from the response to confirm what actually served the request.
Step 5: Implement fallback without thrashing
Even the cheapest model can be rate-limited. Wrap the call in a retry that walks the candidate list in cost order. If you are on a gateway with automatic fallback when a provider is rate-limited or degraded, you can rely on that, but explicit client control avoids surprise upgrades to a premium model.
def call_with_fallback(matrix, constraints, payload, client):
ordered = sorted([(spec["cost_tier"], name) for name, spec in matrix.items()
if spec["vision"] and spec["max_images"] >= constraints["n_images"]])
last_err = None
for _, name in ordered:
try:
return client.chat.completions.create(model=name, messages=build_messages(payload))
except Exception as e:
last_err = e
continue
raise last_errAdd exponential backoff between attempts, but cap at two upgrades. If the second-cheapest is also failing, surface the error to the caller instead of burning tokens on a tier-3 model. Your router’s job is cost predictability, not heroics.
Step 6: Verify the router works
Success means the right model was selected and billed as expected. Write a unit test with a fixed matrix and constraints, and an integration check that asserts the response contains a usage field.
def test_select_cheapest():
m = {"a": {"vision": True, "max_images": 1, "cost_tier": 1, "min_quality": "low"},
"b": {"vision": True, "max_images": 1, "cost_tier": 2, "min_quality": "high"}}
c = {"n_images": 1, "pixels": 100, "complexity": 1}
assert select_model(m, c) == "a"For live verification, send a low-complexity single image and inspect resp.usage. Per-token usage metering should show the cheaper model’s ID in your logs. If you see a premium model, your routing directive was overridden—check gateway settings.
Run a curl smoke test to confirm the wire format:
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-1.5-flash",
"messages": [{"role":"user","content":[
{"type":"image_url","image_url":{"url":"https://example.com/cat.jpg"}},
{"type":"text","text":"describe"}
]}]
}'Check the model in the JSON response. If it echoes gemini-1.5-flash, your directive held.
Operational notes
Route vision requests to cheapest model only after you’ve validated quality on a sample set. Cheap models fail silently on rotated text or non-RGB formats. Keep a human-labeled benchmark and run it in CI when you update the matrix. Competitors offer auto-routing, but their selection is opaque and may optimize for their margins. Owning the matrix lets you swap a model the day its price drops. That’s the difference between a cost line you control and one you inherit.
Step 7: Monitor and iterate
Once live, track two metrics: average cost per vision request and fallback rate. A rising fallback rate on the cheapest tier signals provider degradation—open an issue with the provider or shift its tier up temporarily. Periodically re-pull provider docs; vision support changes monthly. Your router is a small piece of code with outsized leverage on the bill.