Auto-route LangChain to the cheapest model with n4n.ai

This langchain cheapest model routing tutorial shows how to wire LangChain to a single OpenAI-compatible gateway and automatically pick the lowest-cost model that meets your quality bar. You’ll build a small router, plug it into ChatOpenAI, and let the gateway handle provider degradation. No bespoke proxy required.

Step 1: Install dependencies and configure credentials

LangChain split its provider integrations into separate packages in 2024. Use langchain-openai even when targeting a non-OpenAI backend, because the gateway speaks the OpenAI chat protocol.

python -m venv .venv
source .venv/bin/activate
pip install langchain-openai openai python-dotenv requests

Keep secrets out of source control. A .env file loaded at process start is sufficient for local runs; in production inject the same variables from your secret manager.

# .env
N4N_API_KEY=sk-...
GATEWAY_BASE_URL=https://api.n4n.ai/v1

from dotenv import load_dotenv
import os

load_dotenv()
assert os.environ.get("N4N_API_KEY"), "set N4N_API_KEY"

Do not use the legacy langchain.llms.OpenAI class. It lacks the base_url passthrough and the newer message types.

Step 2: Point ChatOpenAI at the gateway

ChatOpenAI accepts base_url and api_key. n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models, with automatic fallback when a provider is rate-limited and per-token usage metering. That single endpoint replaces the tangle of provider SDKs.

from langchain_openai import ChatOpenAI

def make_llm(model: str) -> ChatOpenAI:
    return ChatOpenAI(
        model=model,
        api_key=os.environ["N4N_API_KEY"],
        base_url=os.environ["GATEWAY_BASE_URL"],
        temperature=0.2,
        max_tokens=1024,
        max_retries=1,  # gateway already falls back upstream
        request_timeout=30,
    )

The only variable per call is model. The gateway resolves the provider prefix (openai/, anthropic/, meta-llama/) and routes accordingly. If the resolved provider returns 429 or 5xx, the gateway serves an equivalent model from its pool and meters the actual tokens used.

Step 3: Define a cost-tier model registry

You need a local mapping of model IDs to relative cost and capability. Live prices shift weekly; fetch the catalog from the gateway’s /models endpoint and cache it. The shape is the standard OpenAI models response.

import requests

def fetch_model_ids(base_url: str, api_key: str) -> list[str]:
    r = requests.get(f"{base_url}/models", headers={"Authorization": f"Bearer {api_key}"})
    r.raise_for_status()
    return [m["id"] for m in r.json()["data"]]

# print(fetch_model_ids(os.environ["GATEWAY_BASE_URL"], os.environ["N4N_API_KEY"]))

For routing decisions, a static tier table is enough to start. Use relative cost integers, not fabricated cents.

# model_registry.py
MODEL_TIERS = {
    "meta-llama/llama-3-8b-instruct": {"cost": 1, "ctx": 8192, "code": False},
    "openai/gpt-4o-mini":            {"cost": 2, "ctx": 16384, "code": True},
    "anthropic/claude-3-haiku":      {"cost": 3, "ctx": 200000, "code": True},
    "openai/gpt-4o":                {"cost": 8, "ctx": 128000, "code": True},
}

def cheapest_capable(prefers_code: bool, min_ctx: int) -> str:
    cands = [
        (mid, spec) for mid, spec in MODEL_TIERS.items()
        if spec["ctx"] >= min_ctx and (not prefers_code or spec["code"])
    ]
    if not cands:
        raise ValueError("no model meets constraints")
    return min(cands, key=lambda x: x[1]["cost"])[0]

In production, merge this with the live /models poll so new cheaper models appear without code changes.

Step 4: Implement the routing heuristic

A router should inspect the prompt before spending money. Token length and code syntax are cheap proxies for difficulty. This langchain cheapest model routing tutorial uses a regex and character count; you can later swap in an embedding classifier.

import re

CODE_RE = re.compile(r"```|def |class |import |=>|\&#123;.*\&#125;|SELECT ")

def route_prompt(prompt: str) -> str:
    prefers_code = bool(CODE_RE.search(prompt))
    # short, non-code prompts do not need a large context window
    min_ctx = 8192 if len(prompt) &lt; 4000 else 200000
    if len(prompt) &lt; 200 and not prefers_code:
        return "meta-llama/llama-3-8b-instruct"
    return cheapest_capable(prefers_code, min_ctx)

The opinionated call: a 50-token factual question should never hit a frontier model. The marginal quality gain is zero and the cost multiple is 8x. Route on constraints, not on model fame.

Step 5: Wrap routing in a LangChain Runnable

LangChain favors runnables so the router composes with prompts and output parsers. Below is a synchronous version and an async variant for concurrent workloads.

from langchain_core.runnables import RunnableLambda
from langchain_core.messages import HumanMessage

def invoke_routed(prompt: str) -> dict:
    model = route_prompt(prompt)
    llm = make_llm(model)
    resp = llm.invoke([HumanMessage(content=prompt)])
    return &#123;"content": resp.content, "model_used": model, "usage": resp.usage&#125;

router = RunnableLambda(invoke_routed)

# Async path for FastAPI or asyncio scripts
async def ainvoke_routed(prompt: str) -> dict:
    model = route_prompt(prompt)
    llm = make_llm(model)
    resp = await llm.ainvoke([HumanMessage(content=prompt)])
    return &#123;"content": resp.content, "model_used": model, "usage": resp.usage&#125;

Chain it with a prompt template when you need structured input:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([("user", "&#123;q&#125;")])
chain = prompt | RunnableLambda(lambda d: invoke_routed(d["q"]))
print(chain.invoke(&#123;"q": "What is the capital of Estonia?"&#125;))

Because the gateway performs automatic fallback, you do not wrap make_llm in tenacity retries. One client retry is enough to surface hard errors.

Step 6: Forward cache-control and routing directives

Long system prompts waste tokens on every call. Providers like Anthropic support prompt caching; the gateway forwards cache-control hints passed in extra_body to the upstream without modification. Pass them only when the selected model advertises support.

def invoke_with_cache(prompt: str, system: str) -> str:
    model = route_prompt(prompt)
    llm = ChatOpenAI(
        model=model,
        api_key=os.environ["N4N_API_KEY"],
        base_url=os.environ["GATEWAY_BASE_URL"],
        extra_body=&#123;"cache_control": &#123;"type": "ephemeral", "prefix": system&#125;&#125;,
    )
    return llm.invoke([
        &#123;"role": "system", "content": system&#125;,
        &#123;"role": "user", "content": prompt&#125;,
    ]).content

Client routing directives are honored through the model string itself. To pin a provider, use the fully qualified ID from the registry. To allow the gateway to substitute on degradation, use the same ID and rely on its fallback—your model_used field will reflect the actual served model.

Step 7: Verify success and meter cost

Verification confirms two things: the cheap model was selected for easy prompts, and token usage is reported in the response. Run this script:

from langchain_core.messages import HumanMessage

tests = [
    "Capital of Estonia?",
    "Write a Python function to parse CSV with streaming and type hints.",
]

for t in tests:
    model = route_prompt(t)
    llm = make_llm(model)
    resp = llm.invoke([HumanMessage(content=t)])
    print(f"prompt: &#123;t[:40]!r&#125;")
    print(f"  routed_model: &#123;model&#125;")
    print(f"  actual_model: &#123;resp.response_metadata.get('model', 'n/a')&#125;")
    print(f"  usage: &#123;resp.usage&#125;")

Success criteria:

The first prompt routes to meta-llama/llama-3-8b-instruct (or your tier-1 entry).
The second prompt routes to a code-capable tier-2+ model.
resp.usage.prompt_tokens and completion_tokens are non-zero and match your gateway metering dashboard.

If actual_model differs from routed_model, the gateway applied fallback—expected during provider incidents, not during steady state.

Operational notes

Store MODEL_TIERS in a YAML or JSON config, not in Python source. Prices and context windows change; a config reload should not require a deploy.
Set max_retries=1 on the client. The gateway already handles provider degradation; client-side retry storms only amplify load.
For batch inference, classify the whole batch by its largest prompt rather than per item. Per-item routing adds latency and fragments cache locality.
Cache-control hints are ignored by providers that do not support them; the gateway passes them through regardless, so guard with a capability check from /models.
Log model_used and usage on every call. Per-token metering is only useful if your own dashboards attribute cost to features.

This langchain cheapest model routing tutorial gave you a working pattern: heuristic routing in front of a single OpenAI-compatible endpoint, with the gateway absorbing provider flakiness. Replace the static registry with a live price feed and you have a cost-aware inference layer that degrades gracefully.