How much do LLM API gateways markup provider pricing

Most teams evaluating an LLM API gateway markup assume the penalty is a simple percentage added to the provider’s list price. In practice, the visible surcharge is often small—sometimes zero—but the effective cost delta comes from retry overhead, fallback routing, and cache behavior that direct calls ignore. This analysis breaks down where the markup hides and how to calculate what you actually pay.

The obvious markup: per-token surcharge

Provider list prices are public. OpenAI charges $5 per million input tokens and $15 per million output tokens for GPT-4o. Anthropic lists Claude 3.5 Sonnet at $3 per million input and $15 per million output. A gateway sits in front and resells those tokens.

The explicit LLM API gateway markup is usually expressed as a multiplier on these rates. OpenRouter-class aggregators often publish provider-equivalent pricing with a small add-on—frequently under 5% for commodity models, occasionally 10–20% for niche or fine-tuned endpoints. Some enterprise gateways skip per-token markup entirely and instead impose a flat monthly platform fee or minimum commit.

PROVIDER_PRICE = {"gpt-4o": (5.0, 15.0), "claude-3.5-sonnet": (3.0, 15.0)}  # $/M tok
GATEWAY_MARKUP = 0.05  # 5% LLM API gateway markup

def cost(model, in_tok, out_tok, markup=0.0):
    in_rate, out_rate = PROVIDER_PRICE[model]
    base = in_tok/1e6 * in_rate + out_tok/1e6 * out_rate
    return base * (1 + markup)

If you only read the pricing page, you conclude the gateway costs 5% more. That is the least interesting number in the equation.

Hidden markup: failure and retry overhead

Direct calls fail. Providers return 429s, 500s, and truncated streams. When a request fails after the model has emitted partial output, you either discard those tokens or pay to regenerate them. If your client retries naively, you bill the same prompt prefix again.

Assume a 1% hard failure rate on a 2,000-output-token completion at $15/M. Direct cost per attempt is $0.03. On failure you retry, paying another $0.03. Expected spend per successful completion is $0.03 × (1 + 0.01) = $0.0303. That is a 1% invisible markup from instability alone—before you account for the engineering time to write retry logic, backoff, and idempotency keys.

fail_rate = 0.01
def effective_direct(base, fail_rate):
    return base * (1 + fail_rate)  # ignores partial-output double billing

Now layer a gateway that performs automatic fallback when a provider is degraded. If the fallback cuts the failure rate to 0.1% but applies a 5% surcharge, the expected cost is base * 1.05 * 1.001. The gateway looks more expensive on paper and cheaper in production. The LLM API gateway markup is no longer a single number; it is a function of your traffic shape and the provider’s uptime.

What the markup buys: routing, fallback, cache

A gateway earns its spread by centralizing concerns you would otherwise ship yourself:

• Unified auth and metering. One API key, per-token usage logs, no scattered provider dashboards.
• Provider failover. Automatic reroute on rate limit or degradation.
• Cache-control propagation. Honors cache_control hints so providers bill cached prefix tokens at the discounted rate instead of full price.

Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints, so a prompt prefix cached at the provider is not billed twice when the gateway retries a failed request. That single feature can offset a 5% markup if your workload repeats long system prompts.

Without a gateway, you implement these as custom middleware. The hidden cost is the code you maintain and the incidents you debug at 3 a.m. when a provider changes its error schema.

Comparing direct vs gateway: a concrete example

Take a service processing 10M input and 2M output tokens monthly on GPT-4o.

Direct provider cost:

• Input: 10 × $5 = $50
• Output: 2 × $15 = $30
• Subtotal: $80

Apply 1% failure overhead on output (double-billed retries): +$0.30 → $80.30.

Gateway with 5% markup, fallback reduces failure overhead to 0.1%:

• Marked-up subtotal: $80 × 1.05 = $84.00
• Failure overhead: $84 × 0.001 = $0.084 → $84.08

Gateway with 20% markup, same reliability:

• $80 × 1.20 = $96.00 + $0.096 = $96.10

At 5% the gateway costs ~4.7% more than direct with realistic failures, while removing retry code. At 20% it costs 20% more—you need a concrete feature (compliance logging, dedicated support, multi-region routing) to justify that.

A routing directive in a gateway request might look like:

{
  "route": {
    "prefer": ["anthropic/claude-3.5-sonnet"],
    "fallback": ["openai/gpt-4o"],
    "cache_control": {"type": "ephemeral"}
  }
}

The client states intent; the gateway executes failover without application changes.

How to measure true cost in your stack

Instrument before you trust any markup claim. Log four numbers per request: provider billed input, provider billed output, retries, and wall-clock latency. Compute effective cost per successful response.

curl https://api.example-gateway.ai/v1/chat/completions \
  -H "Authorization: Bearer $GW_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"echo test"}]}'

Parse the usage object. Multiply by your negotiated rate. If you see prompt_tokens_details.cached_tokens populated, your gateway is forwarding cache hints. If not, you are paying full price for repeated prefixes—a silent markup no one put on the pricing page.

Run this against direct provider and gateway side-by-side for a week. The delta is your real LLM API gateway markup, inclusive of reliability.

Tradeoffs and when to skip the gateway

Skip the gateway if:

• You call exactly one provider, at low volume, and its SDK retry behavior is sufficient.
• You have no regulatory need for unified audit logs.
• Your prompt prefixes are tiny, so cache propagation saves nothing.

Use a gateway if:

• You route across three or more providers to manage cost or capability.
• You face sporadic provider degradation that breaks user-facing flows.
• You want per-token metering without building an internal billing system.

The markup is not inherently evil. A 10% surcharge that eliminates a full-time equivalent building fallback logic is a net win. A 30% surcharge for a thin pass-through proxy is tax on inertia.

Takeaway

The LLM API gateway markup printed on a pricing page is a distraction. Compute effective cost: provider list price × (1 + explicit markup) × (1 + failure overhead) − cache savings. For most production systems spanning multiple models, a 5–15% surcharge is cheap insurance and often negative in net cost once you delete your custom retry code. Above 25%, demand concrete features or walk away. Measure with real traffic, not spreadsheets.