Routing requests across GPT-5, Llama 4, and DeepSeek V4

To route requests gpt-5 llama 4 deepseek v4 without maintaining three separate SDKs, you need a single normalized API surface and a deliberate routing policy. This guide walks through the exact integration steps we ship in production, from request shaping to fallback and per-token metering.

1. Normalize the API surface

All three models speak variants of the OpenAI chat completion schema. If you point the OpenAI Python client at a gateway that proxies these models, the only variable that changes is the model string. Do not write per-vendor adapters; that multiplies surface area and breaks on every provider schema tweak.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.example.com/v1",  # your inference gateway
    api_key="sk-...",
)

def chat(model: str, messages: list):
    return client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.2,
    )

# route requests gpt-5 llama 4 deepseek v4 by swapping the model id
chat("gpt-5", [{"role": "user", "content": "Summarize this PR"}])
chat("llama-4", [{"role": "user", "content": "Summarize this PR"}])
chat("deepseek-v4", [{"role": "user", "content": "Summarize this PR"}])

Map internal task names to model IDs in one config file. This keeps call sites agnostic to which backend executes them.

{
  "routes": {
    "fast_summarization": "llama-4",
    "hard_reasoning": "gpt-5",
    "code_review": "deepseek-v4"
  }
}

Pitfall: provider-specific parameters (logit_bias, custom response_format extensions, seed semantics) are not portable. Validate against a schema subset you control, and reject unknown fields before sending.

2. Define a routing policy

Routing is a decision: which model handles which request class. Base it on measurable attributes, not vibes. When you route requests gpt-5 llama 4 deepseek v4 by capability, you avoid overloading expensive models with bulk work they are bad at economically.

Capability-based routing

GPT-5 handles ambiguous multi-step reasoning better. DeepSeek V4 is strong on code synthesis and diff analysis. Llama 4 is cheap for high-volume classification and extraction.

def select_model(task: str) -> str:
    if task in ("planning", "agent_loop"):
        return "gpt-5"
    if task in ("sql_gen", "diff_review"):
        return "deepseek-v4"
    return "llama-4"  # default bulk

Cost and latency tiers

Set a max latency budget per route. If p95 must be under 800 ms, Llama 4 typically wins; GPT-5 may breach under load. Build a shadow eval harness: send 1% of traffic to a candidate model and compare output against a golden set before promoting it.

Tradeoff: cheaper models drift on instruction adherence. A 20% cost save is meaningless if you need three retries. Track task success rate, not just token cost.

3. Implement client-side fallback

Providers fail. When GPT-5 returns 429 or 503, fall back to DeepSeek V4 for non-critical paths. Write explicit exception handling; do not rely on generic catch-alls.

from openai import APIError, RateLimitError

def chat_with_fallback(messages, primary="gpt-5", secondary="deepseek-v4"):
    try:
        return chat(primary, messages)
    except RateLimitError:
        return chat(secondary, messages)
    except APIError as e:
        if e.status_code >= 500:
            return chat(secondary, messages)
        raise

Do not fall back on 400-class errors—those signal a malformed request, not degradation. Common pitfall: fallback chains that retry the same broken model. Cap attempts and emit a metric on each fallback.

Add a circuit breaker per model. If DeepSeek V4 itself is erroring, stop sending it traffic for 30 seconds.

from datetime import datetime, timedelta

breaker = {"deepseek-v4": {"open_until": datetime.min}}

def maybe_breaked(model: str):
    if datetime.now() < breaker[model]["open_until"]:
        raise RuntimeError(f"{model} circuit open")

4. Use gateway-level fallback and cache hints

A gateway that aggregates providers can do automatic fallback when a backend is rate-limited or degraded. n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and honors client routing directives, so you can shift traffic with a header instead of code changes. If your gateway supports similar directives, use them to keep routing logic declarative.

Forward provider cache-control hints to cut repeat prompt costs. With an OpenAI-compatible call, pass extra_headers:

client.chat.completions.create(
    model="gpt-5",
    messages=[{"role": "system", "content": "You are a terse bot"}],
    extra_headers={"x-cache-control": "max-age=3600"},
)

If the gateway forwards these hints, repeated system prefixes hit provider prompt caches. Tradeoff: cache keys are provider-specific. Test that your gateway actually forwards the header; don’t assume.

Also honor client routing directives for sticky sessions. Some gateways let you pin a request to a specific provider region via header. Use that for stateful multi-turn agents.

5. Meter usage and observe

Per-token metering is non-negotiable. Capture usage from each response and tag by route. Without this you cannot compute true cost per task.

resp = chat("llama-4", messages)
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Ship these to your metrics pipeline with labels: model, route, status. Distinguish gateway latency from provider latency—a slow gateway masks a healthy model.

Observability checklist

• Log model ID and gateway latency separate from provider latency.
• Alert on fallback rate > 5% sustained for 10 minutes.
• Track cache hit ratio if the gateway exposes it.
• Record which route selected which model; replay for debugging.

6. Test the routing layer

Integration tests must cover model swap and fallback. Mock the gateway with a local stub that returns 429 for gpt-5 and 200 for deepseek-v4.

def test_fallback(monkeypatch):
    calls = []
    def fake_chat(model, messages):
        calls.append(model)
        if model == "gpt-5":
            raise RateLimitError("no", response=None, body=None)
        return {"model": model}
    monkeypatch.setattr("client.chat", fake_chat)
    out = chat_with_fallback([{"role":"user","content":"hi"}])
    assert out["model"] == "deepseek-v4"
    assert calls == ["gpt-5", "deepseek-v4"]

Run this in CI. A routing bug is silent until a provider dies in production.

Common pitfalls when you route requests gpt-5 llama 4 deepseek v4

• Prompt drift: Same prompt yields different output shapes. Enforce JSON schema validation downstream, not at the model boundary.
• Streaming mismatches: Not all models stream SSE identically. Use the gateway’s normalized stream or buffer complete responses for uniform handling.
• Hidden rate limits: DeepSeek V4 may have lower per-minute quotas than GPT-5. Your fallback must respect that or you cascade failures.
• Version pinning: Model IDs like gpt-5 may alias to newer snapshots. Pin snapshots if reproducibility matters for eval.
• Cache key collisions: Long system prompts with user-specific text break caching. Separate static prefix from dynamic suffix.

Final ordered path

1. Stand up one OpenAI-compatible endpoint for all backends.
2. Map task classes to model IDs in a single config.
3. Write a select_model function based on capability and latency tiers.
4. Add client-side fallback for 429/5xx with circuit breakers.
5. Enable gateway fallback and forward cache-control headers.
6. Meter per token, label by route, and alert on anomalies.
7. Test fallback and routing swaps in CI.

That is the integration pattern we run. It keeps your call sites stable while you route requests gpt-5 llama 4 deepseek v4 across shifting provider health, and it survives the next model drop without a rewrite.