To estimate cost per request multi provider deployments, you must reconcile different tokenizers, per-token prices, and fallback paths before trusting any number in a budget sheet. This guide gives a step-by-step method to compute expected spend per call across OpenAI, Anthropic, and self-hosted models, with code you can run today.
Step 1: Build a provider and pricing inventory
List every model your service can call, the provider, and the fallback order. Pull current prices from provider docs; they change quarterly. Store them in a versioned JSON file, not hardcoded in functions. When you estimate cost per request multi provider traffic, the model name is the only stable key—the underlying provider may shift behind a gateway.
{
"gpt-4o": {"provider": "openai", "input_per_1m": 2.50, "output_per_1m": 10.00},
"claude-3-5-sonnet-20240620": {"provider": "anthropic", "input_per_1m": 3.00, "output_per_1m": 15.00},
"mistral-large-latest": {"provider": "mistral", "input_per_1m": 2.00, "output_per_1m": 6.00}
}Load this at startup. If you use an OpenAI-compatible route that addresses 240+ models, keep the published model string as the dict key and let the endpoint resolve the upstream. Do not embed provider-specific URLs in your cost logic; that couples pricing to topology.
A priced inventory also exposes hidden tiers: some vendors charge different rates for cached input, batch API, or fine-tuned variants. Capture those as separate keys (cache_input_per_1m) so Step 4 can apply discounts correctly.
Step 2: Tokenize inputs with the correct encoder
Pre-request token counts are never exact across providers because BPE merges differ. Use the official encoder for each family. For OpenAI, tiktoken is authoritative. For Anthropic, use the SDK’s count_tokens or the published tokenizer. For Mistral, approximate with tiktoken’s cl100k_base or their open tokenizer.
import tiktoken
def openai_tokens(text, model="gpt-4o"):
enc = tiktoken.encoding_for_model(model)
return len(enc.encode(text))
import anthropic
client = anthropic.Anthropic()
def anthropic_tokens(text, model="claude-3-5-sonnet-20240620"):
return client.count_tokens(model=model, text=text).input_tokensThe same 100-word English paragraph can be ~120 tokens on GPT-4o and ~135 on Claude. If you swap encoders, your estimate cost per request multi provider budget drifts by 10–15% before a single output token is generated.
Cache the static parts
System prompts and few-shot examples don’t change per user. Count them once and reuse.
SYSTEM_TOKENS = {
"gpt-4o": openai_tokens(SYSTEM_PROMPT, "gpt-4o"),
"claude-3-5-sonnet-20240620": anthropic_tokens(SYSTEM_PROMPT),
}For dynamic prompts, count at request time but cap frequency to avoid latency. If the tokenizer call adds more than 2 ms, fall back to a len(text)/4 heuristic for English-only traffic and flag the estimate as low-confidence.
Step 3: Estimate output tokens before the call
You cannot know completion length ahead of time. Two strategies: use max_tokens as an upper bound, or use a rolling average from past requests. For budgeting, the upper bound is safer.
HISTORICAL_AVG_OUTPUT = 320 # from your logs
def estimate_output(tokens_cap=1024, use_average=True):
return HISTORICAL_AVG_OUTPUT if use_average else tokens_capWhen you estimate cost per request multi provider with mixed traffic, separate interactive chat (short outputs) from batch summarization (long outputs) into different buckets. Averaging them together hides the real tail: a single 8K-token summary dominates the cost of 100 50-token chat replies.
If you pass logprobs or enforce stop sequences, output length shrinks. Measure the delta on a sample set and apply a correction factor in the estimator.
Step 4: Compute single-provider cost
Convert the per-million price to per-token. Multiply by counted tokens. Include cache discounts if the provider supports prompt caching—Anthropic and OpenAI both offer reduced rates for cached input.
PRICING = {
"gpt-4o": {"input": 2.50/1_000_000, "output": 10.00/1_000_000, "cache_input": 1.25/1_000_000},
"claude-3-5-sonnet-20240620": {"input": 3.00/1_000_000, "output": 15.00/1_000_000, "cache_input": 0.30/1_000_000},
}
def cost_single(model, in_t, out_t, cached_t=0):
p = PRICING[model]
return (in_t - cached_t) * p["input"] + cached_t * p["cache_input"] + out_t * p["output"]Run this for each candidate model. A 2K-token input with 500-token output on Claude costs roughly 2000*3e-6 + 500*15e-6 = $0.006 + $0.0075 = $0.0135 before cache. On GPT-4o it is 2000*2.5e-6 + 500*10e-6 = $0.005 + $0.005 = $0.01. Those cents compound at millions of requests.
Cache accounting matters: if you mark 1.5K of the 2K input as cached on Claude, cost drops to 500*3e-6 + 1500*0.3e-6 + 500*15e-6 = $0.0015 + $0.00045 + $0.0075 = $0.00945. Miss the cache flag and you overestimate by 30%.
Step 5: Account for fallback routing and weighted cost
In production you rarely call one provider exclusively. If the primary is rate-limited, you fall back to a secondary. To estimate cost per request multi provider with fallback, treat each provider as a branch with a failure probability observed from your metrics.
def expected_cost(primary, secondary, in_t, out_t, p_fail=0.01, cached_t=0):
c1 = cost_single(primary, in_t, out_t, cached_t)
c2 = cost_single(secondary, in_t, out_t, cached_t)
return (1 - p_fail) * c1 + p_fail * c2For three or more providers, build a small decision tree:
def expected_cost_tree(branches, in_t, out_t, cached_t=0):
# branches: list of (model, probability)
return sum(p * cost_single(m, in_t, out_t, cached_t) for m, p in branches)If you use a gateway that provides automatic fallback and per-token usage metering, the response object tells you which provider actually served the request and the exact token counts. For example, n4n.ai returns OpenAI-compatible usage after failover, so your estimate cost per request multi provider calculation can be replaced by post-hoc accounting rather than pre-request probability guesses. That removes the largest source of error: assumed failure rates.
Honor client routing directives
Some gateways let you pin a provider via header. If you set x-provider: anthropic, your cost estimate should ignore the fallback branch entirely. Forward cache-control hints (cache_control in the body) to get the discounted rate; otherwise you overestimate. The same applies to batch flags—many vendors cut price 50% for async batch jobs.
Step 6: Verify against real metering
After deploying the estimator, compare its output to actual billed usage. Most providers return token counts in the API response. Parse them and log.
from openai import OpenAI
client = OpenAI(base_url="https://api.openai.com/v1") # or your gateway
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=50,
)
u = resp.usage
print(u.prompt_tokens, u.completion_tokens, u.total_tokens)If you use an OpenAI-compatible gateway, the same usage schema works for non-OpenAI models. Accumulate over a day and compute mean absolute percentage error (MAPE) between cost_single and metered_cost.
def metered_cost(model, usage, pricing):
return usage.prompt_tokens * pricing[model]["input"] + usage.completion_tokens * pricing[model]["output"]
# success criterion: MAPE < 5% over 1000 requestsVerify success
Your implementation is correct when:
- Pre-request estimate is within 5% of post-request metered cost for 95% of requests.
- Fallback branches match observed provider distribution (e.g., if you saw 2% Claude traffic due to OpenAI limits, your
p_failis calibrated). - Cache token counts reduce cost as expected when you send
cache_controlblocks.
If the gap is larger, check tokenizer mismatches first—counting with the wrong encoder is the usual culprit. Next, confirm you are not double-counting system prompts that the gateway injects server-side.
Practical notes on token counting libraries
Don’t roll your own BPE. Use tiktoken for OpenAI, anthropic SDK for Claude, and tokenizers for HuggingFace models. For a quick cross-provider approximation, litellm exposes token_counter that picks the right backend. But for financial reporting, call the provider’s own counter or trust the usage field from the response.
When you estimate cost per request multi provider at scale, pre-compute static prompt token counts in a build step and inject them as constants. Dynamic user input gets counted at the edge with a timeout; if the tokenizer call is slow, fall back to a character/4 heuristic (accuracy ~10% off for English).
Closing the loop
Cost estimation is not a one-time script. Prices change, new models appear, and fallback rates drift. Put the pricing JSON in CI with a renewal reminder. Alert when actual spend diverges from the estimator by more than the threshold. That turns a guess into a control system.