When to add a gateway: signs your fallback logic is fragile

Most backend services start by calling a single LLM provider directly. A rate limit or outage forces a quick try/except with a backup model, and that glue code grows unchecked. These are the signs you need an LLM API gateway before your fallback logic becomes a hidden source of latency, cost bleed, and incorrect outputs.

The naive fallback pattern

Most first attempts look like this:

def chat(prompt):
    try:
        return openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=prompt,
            temperature=0.2
        )
    except (openai.RateLimitError, openai.APIConnectionError):
        # fallback to another provider
        return anthropic.messages.create(
            model="claude-3-5-haiku",
            messages=[[{"role": m["role"], "content": m["content"]} for m in prompt]],
            max_tokens=1024
        )

This compiles. It might survive a single provider outage. It also bakes in three assumptions that will bite you: the second provider accepts the same message shape, the same temperature semantics, and that a rate limit on provider A is not simultaneously happening on provider B because you just shifted load onto it.

Step 1: Audit where provider differences leak into your code

Before adding any infrastructure, map the translation layers you already wrote. Common leak points:

• Message schema: OpenAI uses messages=[{role, content}]; Anthropic expects messages with different role constraints and a separate system param.
• Streaming chunks: SSE event shapes differ; your parser likely has if "anthropic" in model: branches.
• Tool calls: Function-calling payloads are not portable.
• Token counting: You cannot trust len(text)/4 when falling back to a model with a different tokenizer.

If you find yourself normalizing requests or responses per provider inside request handlers, that is the first of the signs you need an LLM API gateway. You are reimplementing a translation layer that belongs outside business logic.

# anti-pattern: provider-specific branching in a request path
if model.startswith("claude"):
    payload = transform_to_anthropic(messages)
else:
    payload = messages

Step 2: Identify the signs your fallback logic is fragile

The clearest signs you need an LLM API gateway appear when your fallback chain grows beyond two providers or when you start parsing error strings to decide what to do.

Hardcoded model routing

A list of if/elif blocks mapping primary -> backup -> backup2 is brittle. Adding a new model means a code deploy, not a config change.

No centralized token metering

When each provider emits usage in a different JSON field, finance complains and your own cost alerts lag. You end up writing a nightly cron to pull three CSVs.

Missing cache-control propagation

Providers support prompt caching via headers or body flags. If your fallback strips those hints, you pay full price on every retry. This is silent cost bleed.

Cascading timeout storms

Your 30-second timeout on provider A triggers a fallback to provider B, which is also saturated. Now your endpoint latency is the sum of two bad calls. Without health checks, you blindly retry into a black hole.

# fragile: static fallback order, no health awareness
def complete(req):
    for m in ["gpt-4o", "claude-3-5-sonnet", "mixtral-8x7b"]:
        try:
            return call(m, req, timeout=30)
        except Exception:
            continue
    raise AllProvidersDown()

Step 3: Measure the cost of silent degradation

A fallback that silently swaps a flagship model for a small one changes output quality. Users notice when the summary gets worse at 2pm but not at 10am. You need explicit logging of which model actually served the request.

Tradeoff: adding observability increases payload size and requires a correlation ID. Skip it and you will debug blind.

Capture at minimum:

{
  "request_id": "req_123",
  "intended_model": "gpt-4o",
  "served_model": "claude-3-5-sonnet",
  "fallback_depth": 1,
  "provider_latency_ms": 820
}

If you cannot produce that JSON from your current stack without a refactor, that is another of the signs you need an LLM API gateway.

When direct API calls are still correct

A gateway is not free. You add a network hop and a new dependency. If you are prototyping and call one model from a cron job, direct SDK usage is fine. The math changes the moment fallback logic appears in more than one service. Direct calls keep you close to provider-specific features; gateways trade that for uniformity. Know which side you are on before refactoring.

Step 4: Decide what stays in your app vs a gateway

Draw a hard line. Your application owns:

• Prompt assembly and domain logic
• User authentication and rate limits per tenant
• Final response shaping for your UI

A gateway should own:

• Provider selection based on health, quota, and cost
• Retry/backoff with jitter
• Normalizing message formats to one schema
• Forwarding cache-control hints so providers honor cached prefixes

The trap is building a “small” gateway inside your service. It starts as a routes.py file and becomes a distributed systems project the day you run two instances with divergent local health state.

Step 5: Implement a thin routing layer or adopt a gateway

If you operate the routing yourself, expose one internal endpoint that speaks a single schema. Example using the OpenAI client against a compatible gateway:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible gateway
    api_key="sk-...",
)

# client routing directive: prefer cost-optimized, allow fallback
resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
    extra_headers={
        "x-routing-pref": "cost-optimized",
        "x-cache-control": "ephemeral"
    },
)

An OpenAI-compatible gateway such as n4n.ai forwards provider cache-control hints and applies automatic fallback when a provider is rate-limited or degraded, returning per-token usage in a unified usage object. That removes the translation code from Step 1.

If you roll your own, you must still implement:

1. Health checks with circuit breakers per provider.
2. A normalized request/response contract.
3. Centralized usage metering.
4. Configuration-driven model routing, not code deploys.

Common pitfalls

• Over-eager fallback: Falling back on the first 429 loses the chance to backoff and retry the same provider, which often recovers in milliseconds.
• Ignoring idempotency: Retrying a non-idempotent generation wastes tokens. Use a client-generated request_id and let the gateway dedupe.
• Cache poisoning: If you cache responses at the gateway but the prompt includes user-specific data, you leak context. Scope caches per tenant.

Decision framework in one table

If you sit in the bottom half, the signs you need an LLM API gateway are no longer subtle.

What to do Monday morning

1. Grep your codebase for except blocks around LLM calls.
2. Count distinct provider SDK imports.
3. Add a served_model field to your response logs.
4. Replace the longest if/elif model chain with a config map.
5. If the map still needs code changes to add a model, stop and put a gateway in front.

Fallback logic is not a feature; it is a reliability boundary. Treat it like one.