Most teams evaluate LLM infrastructure on published per-token rates and ignore the hidden fees LLM API gateways stack on top. Those fees show up as markup spreads, cache penalties, and fallback surcharges that quietly inflate monthly spend by double digits.
The markup spread: token resale economics
A gateway sits between your code and the model provider. It buys tokens at the provider rate and resells them to you. The difference is its margin. Some gateways disclose a flat percentage. Others embed the markup in the model price table, so a model appears at $4.50 per million input tokens when the provider charges $3. That $1.50 is a hidden fee unless the gateway documents the spread.
The dangerous variant is the dynamic markup. Your request goes to whichever provider has capacity; the gateway charges a blended rate that can shift daily. Without per-request cost metadata, you cannot attribute spend.
Many gateways also add a fixed per-request fee on top of the token margin. For high-QPS, low-token workloads (e.g., classification calls averaging 20 tokens), a $0.0001 per-request fee dominates the token cost. Over 10 million calls, that is $1,000 of pure overhead unrelated to model inference.
from openai import OpenAI
client = OpenAI(base_url="https://gateway.example.com/v1", api_key="sk-...")
resp = client.chat.completions.create(
model="claude-3-5-sonnet",
messages=[{"role": "user", "content": "Summarize this log"}]
)
print(resp.usage)
# If gateway markup is hidden, usage.cost is absent; you only see tokens.If the response object lacks a cost or provider_price field, you are flying blind. A legitimate gateway returns the provider’s native usage plus its own margin annotation.
Cache penalties: when your cache_control is silently dropped
Providers like Anthropic and OpenAI offer prompt caching. You send cache_control hints; they store the prefix and bill cached tokens at a discount (often a fraction of full price). A gateway that does not forward those hints destroys the discount.
This is a hidden fee LLM API gateways impose by omission. Your code sets the hint, the gateway strips it, the provider computes the full prompt, and you pay the full rate on that prefix every call.
{
"model": "claude-3-5-sonnet",
"messages": [
{"role": "system", "content": "You are a parser.", "cache_control": {"type": "ephemeral"}}
]
}If the gateway forwards the hint, the provider returns prompt_tokens_details.cached_tokens. If not, that field is zero. Worse, some gateways implement their own intermediate cache and charge storage fees while still not passing the hint to the upstream provider.
A gateway that honors client routing directives and forwards provider cache-control hints (like n4n.ai) avoids this penalty class. But many do not, and their docs omit the limitation. You discover it only when the invoice arrives.
Fallback surcharges: degraded providers cost more than you think
When a primary provider is rate-limited, a gateway may fail over to a secondary. If the secondary is a more expensive model class (e.g., a frontier model instead of a haiku-class one), your per-token cost can jump an order of magnitude. The gateway may not flag the substitution in the response.
Worse, some gateways add a “reliability fee” on fallback events. You pay extra for the privilege of not getting an error.
curl https://gateway.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "x-route-preference: anthropic:claude-3-haiku, openai:gpt-4o-mini" \
-d '{"model":"claude-3-haiku","messages":[{"role":"user","content":"hi"}]}'If the gateway honors x-route-preference, you control cost. If it silently upgrades to a larger model on an outage, inspect the x-routed-model response header. Absent that header, you cannot prove the swap. The fallback surcharge is invisible until you correlate latency spikes with billing line items.
Metering gaps: rounding, batching, and token counting
Tokenizers differ. A gateway using a different tokenizer than the provider will report token counts that diverge from the provider’s bill. The gateway meters you on its count; the provider charges on theirs. The gap is the hidden fee.
Batching is another leak. Some gateways combine multiple short requests into one provider call and split the token count unevenly, or round up to the nearest 4 tokens per request. Over millions of calls, the rounding tax adds up.
Streamed completions exacerbate this: some gateways count tokens on the flushed text using a local tokenizer that disagrees with the provider’s final count by 1–2 tokens per streamed chunk. Multiply by volume.
# Compare gateway usage vs provider usage for identical input
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
local_tokens = len(enc.encode("Same prompt text"))
# If gateway reports local_tokens + 2 consistently, that's a rounding skim.How to audit a gateway bill
You need three artifacts: the gateway’s usage object, the provider’s raw invoice, and the request/response headers.
Parse the usage object
Require usage to include prompt_tokens, completion_tokens, and prompt_tokens_details if caching applies. Log the x-request-id and any x-routed-model.
import logging
logging.info("req=%s model=%s used=%s",
resp.headers.get("x-request-id"),
resp.headers.get("x-routed-model"),
resp.usage.model_dump())Replay against provider directly
For a sample of 100 requests, send the same payload to the provider’s native endpoint. Compare token counts and cached token fractions. Any systematic delta is a fee. Compute effective price per token from your gateway invoice and compare to the provider’s published rate.
Monitor routing headers
If the gateway supports client routing directives, set them and assert the response header matches. n4n.ai provides per-token usage metering that matches provider counts, which simplifies this check. Build a daily job that flags any response where x-routed-model differs from your preference without an explicit fallback policy.
Tradeoffs: what you get for the fee
Gateways earn their cut by providing unified authentication, fallback, and model abstraction. If you only call one provider, bypass the gateway and pay wholesale. If you need multi-provider redundancy, a transparent gateway is worth a disclosed 5–10% margin.
The problem is opacity. A gateway that hides the markup, drops cache hints, and silently upgrades models converts a convenience fee into a tax on every token. The engineering cost of building your own fallback logic is real, but the alternative is unbounded spend drift.
Decisive takeaway
Treat any gateway that does not expose x-routed-model, forward cache_control, and return provider-aligned token counts as a liability. Audit monthly: replay 1% of traffic to direct providers, diff the usage, and multiply deltas by volume. The hidden fees LLM API gateways charge are recoverable only if you measure them. Choose infrastructure that makes the bill legible, or build the audit harness yourself before the spend grows.