How n4n load balances across 240+ models and providers

Load balancing LLM inference isn’t like balancing web servers. Models have different capabilities, pricing, latency profiles, and failure modes. A request for GPT-4o might route to OpenAI, Azure, or a third-party host — each with distinct rate limits, regional availability, and SLA characteristics. n4n load balancing models providers solves this by treating the model catalog as a unified routing surface with per-request policy enforcement, automatic failover, and usage metering that works the same whether you’re calling one model or two hundred.

The routing layer sits in front of every provider

The gateway receives an OpenAI-compatible request, extracts the model identifier, and resolves it to an ordered list of candidate endpoints. Each candidate carries metadata: provider name, region, supported parameters, pricing tier, and current health state. The router doesn’t hardcode model-to-provider mappings; it reads them from a versioned catalog that updates without gateway restarts.

# Simplified routing resolution
class ModelRouter:
    def __init__(self, catalog: ModelCatalog, health: HealthRegistry):
        self.catalog = catalog
        self.health = health

    def resolve(self, model_id: str, directives: RoutingDirectives) -> list[Endpoint]:
        candidates = self.catalog.endpoints_for(model_id)
        # Filter by client directives (region, provider, cost ceiling)
        candidates = [e for e in candidates if directives.allows(e)]
        # Sort by health score, then latency, then cost
        candidates.sort(key=lambda e: (
            -self.health.score(e.provider),
            e.avg_latency_ms,
            e.cost_per_1k_tokens
        ))
        return candidates

The catalog lives in a separate control plane. When a provider adds a new model or deprecates an endpoint, the catalog updates and the gateway picks up the change within seconds. No deployment required.

Health checks must be model-aware

A provider’s /health endpoint tells you the API is up. It doesn’t tell you whether gpt-4o is returning 500s, whether claude-3-opus is timing out, or whether a specific region is degraded. The gateway runs synthetic requests against each model-endpoint pair on a rolling schedule — lightweight completions with max_tokens=1 — and feeds results into a per-model health score.

class ModelHealthChecker:
    async def check(self, endpoint: Endpoint, model: str) -> HealthSignal:
        start = time.monotonic()
        try:
            resp = await endpoint.complete(
                model=model,
                messages=[{"role": "user", "content": "ping"}],
                max_tokens=1,
                timeout=5.0
            )
            latency_ms = (time.monotonic() - start) * 1000
            return HealthSignal(
                healthy=resp.status == 200,
                latency_ms=latency_ms,
                error_code=None
            )
        except Exception as e:
            return HealthSignal(
                healthy=False,
                latency_ms=5000,
                error_code=type(e).__name__
            )

Health scores decay exponentially. A single failure drops the score; sustained success recovers it. The router uses these scores as weights, not hard gates — a degraded endpoint still receives traffic if alternatives are worse, but at reduced volume.

Pitfall: Checking only the provider root endpoint. You’ll route traffic to a healthy API serving a broken model. Check the model.

Client directives override default routing

Applications often need control: “never use Provider X for PII data,” “prefer EU regions,” “cap cost at $0.01/1k tokens.” The gateway honors x-n4n-routing headers (or query parameters) that express these constraints. Directives are declarative, not imperative — the client states requirements, the router satisfies them.

POST /v1/chat/completions
Authorization: Bearer sk-...
Content-Type: application/json
x-n4n-routing: region=eu, max_cost_per_1k=0.015, exclude=provider:anthropic

{
  "model": "gpt-4o",
  "messages": [...]
}

The router filters candidates before sorting. If no endpoint satisfies the directives, the request fails fast with a 400 explaining which constraint couldn’t be met — not a generic 502 after a timeout.

Tradeoff: Directive complexity. Every added constraint reduces the candidate pool and increases the chance of “no eligible endpoint.” Keep directives minimal; use them for hard requirements (compliance, budget), not preferences.

Fallback is automatic but configurable

When the primary endpoint fails — timeout, 5xx, rate limit (429), or circuit breaker open — the gateway retries the next candidate in the sorted list. Retry policy is per-model and per-error-class:

# Fallback policy example
models:
  gpt-4o:
    retry_on:
      - timeout
      - 5xx
      - 429
    max_retries: 2
    retry_backoff_ms: 200
    circuit_breaker:
      failure_threshold: 5
      recovery_timeout_sec: 30

The gateway preserves the original request ID across retries so upstream providers can deduplicate. It also forwards x-request-id to every provider, enabling end-to-end tracing.

Pitfall: Blind retries on 400-class errors. A malformed request will fail identically on every provider. Only retry on transient failures (timeout, 5xx, 429). The gateway classifies error codes automatically but exposes the policy for overrides.

Streaming requires special handling

Streaming responses complicate fallback. If the first chunk arrives successfully but the stream breaks mid-way, you can’t transparently switch providers — the client has already received partial output. The gateway handles this by:

1. Buffering the first N chunks (configurable, default 3) before releasing to the client
2. If the stream fails during buffering, failing over silently and restarting the request
3. If the stream fails after release, returning an error chunk and closing — the client must handle reconnection

async def stream_with_fallback(request, candidates):
    for endpoint in candidates:
        buffer = []
        try:
            async for chunk in endpoint.stream(request):
                buffer.append(chunk)
                if len(buffer) >= BUFFER_THRESHOLD:
                    # Flush buffer, then stream live
                    for c in buffer:
                        yield c
                    buffer = []
                    async for c in endpoint.stream(request):
                        yield c
                    return  # Success
        except TransientError:
            continue  # Try next candidate
        except Exception:
            if buffer:
                # Partial success — can't retry cleanly
                yield error_chunk("stream interrupted")
            raise
    raise NoHealthyEndpointError()

This adds ~50-100ms latency for the buffering window but prevents partial outputs reaching clients.

Usage metering works across providers

Each provider reports usage differently: OpenAI returns usage in the final chunk, Anthropic includes it in every streaming chunk, some third parties omit it entirely. The gateway normalizes this into a canonical Usage object attached to every response (or final streaming chunk).

@dataclass
class NormalizedUsage:
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    provider: str
    model: str
    estimated_cost_usd: float
    cached_tokens: int = 0  # When provider reports cache hits

Cost estimation uses the gateway’s pricing table, which maps (provider, model) → (input_price, output_price) per 1k tokens. Prices update via the same catalog mechanism as endpoints. The normalized usage enables:

• Per-request cost headers (x-n4n-usage-cost-usd)
• Aggregated billing dashboards
• Budget enforcement at the API key level

Pitfall: Trusting provider-reported token counts for billing without verification. Some providers count differently (e.g., including system prompt tokens, excluding cached tokens). The gateway logs both provider-reported and locally-counted tokens for reconciliation.

Cache-control hints flow through

Providers increasingly support prompt caching (OpenAI, Anthropic, Google). The gateway forwards Cache-Control headers from provider responses to clients, and respects Cache-Control: no-store on requests. It also exposes a x-n4n-cache-status header indicating hit, miss, or bypass.

# Response headers
x-n4n-cache-status: hit
cache-control: max-age=3600, private
x-n4n-usage-cost-usd: 0.00042

This lets clients build cache-aware logic (e.g., “retry with no-store on cache miss”) without parsing provider-specific headers.

Observability: know what actually happened

The gateway emits structured logs for every request, including the routing decisions:

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "request_id": "req_abc123",
  "model": "gpt-4o",
  "routing": {
    "candidates_considered": 4,
    "selected": "azure-eastus",
    "directives_applied": ["region=us"],
    "fallback_attempts": 0
  },
  "latency_ms": 847,
  "status": 200,
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 340,
    "cost_usd": 0.018
  }
}

Key fields to alert on:

• fallback_attempts > 0 — primary endpoint degraded
• candidates_considered == 1 — no redundancy for this model
• routing.selected shifting over time — capacity changes

Common pitfalls and how to avoid them

1. Treating all models as interchangeable. gpt-4o and claude-3-5-sonnet have different strengths. Routing purely by latency/cost ignores quality. Solution: tag models with capability vectors (reasoning, coding, multilingual, context window) and let clients specify minimum capabilities via directives.

2. Ignoring regional data residency. Routing a EU user’s request to a US endpoint violates GDPR. Solution: make region a required directive for regulated workloads; the gateway rejects requests that can’t satisfy it.

3. Over-retrying rate-limited providers. Hammering a 429’ing endpoint wastes quota and delays the request. Solution: respect Retry-After headers, implement token-bucket rate limiting per provider in the gateway, and back off exponentially.

4. No visibility into provider SLA drift. A provider’s latency creeps from 800ms to 3s over weeks. Solution: alert on p95 latency per (provider, model) pair with a 7-day baseline.

5. Assuming fallback is free. Each retry adds latency and cost. Solution: track fallback_attempts in metrics; if >10% of requests fall back, investigate the primary.

Start small, expand deliberately

You don’t need 240 models on day one. Begin with:

1. Two providers for your primary model (e.g., OpenAI + Azure for GPT-4o)
2. Health checks on both
3. Directive-based routing for region and cost
4. Structured logging with routing metadata

Add models when you have a concrete use case — a cheaper model for classification, a longer-context model for document QA, a specialized coding model. The catalog handles the rest.

The gateway’s job is to make the complexity of a heterogeneous model fleet invisible to your application code. You send a request to one endpoint; the gateway handles the routing, the fallbacks, the metering, and the observability. Your code stays clean.