Most teams evaluating an inference layer fixate on model quality and latency, but the accounting model decides their burn rate. The core distinction in llm gateway billing tokens vs requests is that tokens reflect actual compute consumed while requests are a convenience tax for connection overhead. Understanding which levers your provider pulls prevents ugly surprises when traffic scales.
The fundamental unit: tokens, not calls
Most inference providers derive cost from the transformer forward pass. That math is linear in token count, not in HTTP requests. A single request that streams 2,000 output tokens costs roughly 20x a request that yields 100 tokens, even if both hit the same endpoint once.
Why tokens map to compute
The GPU memory and FLOPs scale with sequence length. Prompt processing (prefill) and generation (decode) both bill per token. Input tokens are cheaper because they are processed in parallel; output tokens are serial and thus pricier. This is why every OpenAI-compatible response returns a usage object breaking them out.
{
"usage": {
"prompt_tokens": 128,
"completion_tokens": 256,
"total_tokens": 384
}
}Tokenization itself is model-specific. A word like “running” might be one token in GPT-4o but two in Llama-3. A gateway that abstracts many models must normalize metering to the provider’s own count, not guess. The llm gateway billing tokens vs requests clarity starts with trusting the usage block from the upstream model, not the gateway’s estimate.
What a token actually costs
Prices are quoted per million tokens. A gateway that resells model access marks this up. The llm gateway billing tokens vs requests question becomes acute when the gateway adds a separate request fee on top of passthrough token cost. Input tokens typically run at a fraction of output token price; that asymmetry is where most budgets blow up because logging and retry logic silently inflate output.
Where request-based fees enter
Request fees are a flat charge per API call, independent of token volume. They cover connection handling, auth, routing, and sometimes a margin for sporadic low-volume users.
Minimum per-request charges
Some providers impose a minimum billing increment. If your request processes 10 tokens but the meter rounds to 100, you effectively pay a request-style floor. This is common in self-hosted setups fronted by a gateway. The fee is invisible until you slice the invoice by request_id.
Gateway overhead and routing directives
When you send a routing directive like {"route": "anthropic"} or cache-control hints, the gateway may count a request even if it returns an error. Honoring client routing directives is necessary for control, but each attempted call can incur a fee.
curl https://api.example-gateway.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{
"model": "auto",
"messages": [{"role":"user","content":"Hi"}],
"route_hint": "openai"
}'If the route fails and falls back, you might see two requests on the invoice but only one successful token stream. That’s not a bug; it’s a business model.
Reading the metering from the API
Engineers should parse usage programmatically and reconcile against billing statements. Below is a minimal Python snippet that computes cost given a price table and flags request fees.
PRICING = {
"gpt-4o": {"in": 5.0, "out": 15.0}, # per 1M tokens
}
def cost(model, usage, request_fee=0.0):
p = PRICING[model]
token_cost = (usage["prompt_tokens"]/1e6)*p["in"] + (usage["completion_tokens"]/1e6)*p["out"]
return token_cost + request_fee
# from response
usage = {"prompt_tokens": 128, "completion_tokens": 256}
resp_request_fee = 0.0001 # from gateway field
print(f"${cost('gpt-4o', usage, resp_request_fee):.4f}")The llm gateway billing tokens vs requests nuance appears when the gateway response includes a request_id and a separate request_cost field. That field is the red flag.
{
"usage": {"prompt_tokens": 128, "completion_tokens": 256},
"request_cost": 0.0001,
"request_id": "req_abc"
}If you aggregate 100,000 calls, that sub-cent fee becomes a line item larger than your token spend on small prompts.
Hidden mechanics: caching and fallback
Caching changes the token equation. Provider cache-control hints let you mark prompt prefixes as reusable. A gateway that forwards those hints can slash prompt token charges on repeated calls.
Cache-control hints and their billing impact
If you send cache_control: {"type": "ephemeral"} on a system prompt, Anthropic bills cached input tokens at 10% of base rate; OpenAI applies a 50% discount on cache hits. The gateway must pass this through untouched.
resp = client.chat.completions.create(
model="claude-3-5-sonnet",
messages=[{"role":"system","content":"You are a math tutor","cache_control":{"type":"ephemeral"}}],
)Failure to forward the hint means you pay full price silently. That’s a billing leak that no token count will reveal because the tokens look identical.
Provider degradation and automatic fallback
When a primary provider is rate-limited, a mature gateway retries against a secondary. n4n.ai does this across 240+ models with per-token usage metering, so you aren’t double-billed for the failed attempt if it returns no tokens. But not all gateways are honest: some count the failed request as a billed event.
The llm gateway billing tokens vs requests distinction matters here: token billing is naturally idempotent to failures (no tokens, no charge), while request billing punishes you for network weather.
Tradeoffs: predictability vs granularity
Request billing is easy to model: requests * flat_fee. Token billing requires sampling real traffic. Both have merits.
Request billing simplifies forecasting
If your calls are uniform in size, a flat fee per request gives finance a clean line item. Early-stage prototypes with chatty small prompts benefit because the token math is noise.
Token billing exposes inefficiency
Per-token metering shows exactly which feature burns cash. A verbose system prompt repeated every call shows up as a recurring input token tax. You can then cache or trim it. Request fees mask this by flattening the curve.
Decision framework for engineers
Treat request fees as a fixed cost to minimize, token cost as variable to optimize.
How to instrument your client
Log every response’s usage block and the gateway’s request cost. Aggregate by endpoint and model.
interface GatewayUsage {
prompt_tokens: number;
completion_tokens: number;
request_cost?: number;
}
function logUsage(tag: string, u: GatewayUsage) {
console.log(`${tag}: tokens=${u.prompt_tokens + u.completion_tokens} reqfee=${u.request_cost ?? 0}`);
}Pipe this to Prometheus. Alert when request_cost sum exceeds 5% of token cost.
What to negotiate in contracts
If you commit to volume, push for request fee waivers above a threshold. Insist on token-only billing for error responses. The llm gateway billing tokens vs requests debate should end with: errors are free, tokens are metered.
Takeaway
Token-based billing is the only model that tracks real work; request fees are a legacy tax on connection overhead that survives because it simplifies provider accounting. Build your cost dashboards around token streams, treat any per-request line as a negotiable overhead, and choose gateways that meter only what the model produced. If a gateway can’t show you a zero token charge on a failed call, it’s billing you for air.