Building a cost estimator for multi-model LLM apps

Running the same prompt across GPT-4o, Claude, and Mixtral turns cost forecasting into a spreadsheet exercise. To regain control, you should build llm cost estimator tooling that converts token counts into expected spend per provider and per route. This tutorial walks through a small Python library that does exactly that, with hooks for real usage reconciliation against your gateway’s metering.

Prerequisites

• Python 3.10+ installed locally
• pip install tiktoken openai
• A JSON file of model prices (we define the schema; populate it with your providers’ current rate cards)
• An OpenAI-compatible endpoint to call (we’ll use a placeholder base URL)
• Basic comfort with Python classes and JSON

If you already route through a unified gateway, keep your API key handy. The code below is provider-agnostic.

Step 1: Define a pricing schema

Store prices as cost per 1,000 tokens, split by direction. This matches how every major LLM provider bills.

{
  "gpt-4o": {"input": 0.005, "output": 0.015},
  "claude-3-opus": {"input": 0.015, "output": 0.075},
  "mixtral-8x7b": {"input": 0.0006, "output": 0.0006}
}

The numbers above are illustrative placeholders. Replace them with the live prices from each provider’s billing documentation before relying on the output. The schema is deliberately flat so you can load it without transformation.

Step 2: Token counting that degrades gracefully

OpenAI models have official tokenizers via tiktoken. Anthropic and open-weight models do not ship a local encoder you can pip install reliably, so we fall back to a character heuristic. The goal is to never crash a budget job because a model name is unknown.

import tiktoken

def count_tokens(text: str, model: str) -> int:
    try:
        enc = tiktoken.encoding_for_model(model)
        return len(enc.encode(text))
    except KeyError:
        # Non-OpenAI model: rough approximation, 4 chars/token
        return max(1, len(text) // 4)

Checkpoint:

print(count_tokens("Hello world", "gpt-4o"))
print(count_tokens("Hello world", "claude-3-opus"))

Expected output:

3
2

The heuristic returns 2 because "Hello world" is 11 characters, integer-divided by 4. Good enough for an estimate; we’ll correct it with real usage later.

Step 3: Single-model cost estimation

Wrap the pricing and token logic in a class. Keeping it stateful makes batch jobs cleaner.

import json

class CostEstimator:
    def __init__(self, pricing_path: str):
        with open(pricing_path) as f:
            self.pricing = json.load(f)

    def estimate(self, model: str, input_text: str, output_tokens: int) -> float:
        if model not in self.pricing:
            raise ValueError(f"No pricing for {model}")
        price = self.pricing[model]
        in_tok = count_tokens(input_text, model)
        cost = (in_tok / 1000) * price["input"]
        cost += (output_tokens / 1000) * price["output"]
        return round(cost, 6)

Run it:

est = CostEstimator("prices.json")
print(est.estimate("gpt-4o", "Summarize this log", 500))

With the placeholder JSON, input tokens are ~4, so cost is (4/1000)*0.005 + (500/1000)*0.015 = 0.00002 + 0.0075 ≈ 0.00752. The printed value will be 0.00752.

Step 4: Blended cost across a fallback route

Production apps rarely call one model. They try a primary, then fall back to a cheaper or more available one. To build llm cost estimator logic that reflects a real routing policy, compute expected cost as a weighted sum across the chain.

def estimate_route(est: CostEstimator, route: list[str], weights: list[float], input_text: str, output_tokens: int) -> float:
    assert abs(sum(weights) - 1.0) < 1e-6, "weights must sum to 1"
    total = 0.0
    for model, w in zip(route, weights):
        total += w * est.estimate(model, input_text, output_tokens)
    return round(total, 6)

Example: 80% traffic to GPT-4o, 20% to Claude due to rate limits.

route = ["gpt-4o", "claude-3-opus"]
weights = [0.8, 0.2]
print(estimate_route(est, route, weights, "Summarize this log", 500))

This returns a blended figure. If you route through a single OpenAI-compatible endpoint that fronts many models—n4n.ai does this for 240+ models with automatic fallback and per-token usage metering—you can replace the weights with observed traffic ratios from your access logs.

Step 5: Reconcile estimates with real API usage

Estimates are useless if they drift. Call the chat endpoint and use the returned usage object to compute actual cost. This closes the loop.

from openai import OpenAI

client = OpenAI(base_url="https://api.your-gateway.example/v1", api_key="YOUR_KEY")

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this log"}],
    max_tokens=500,
)
usage = resp.usage
price = est.pricing["gpt-4o"]
actual_cost = (usage.prompt_tokens / 1000) * price["input"] \
            + (usage.completion_tokens / 1000) * price["output"]

print(f"Estimated: {est.estimate('gpt-4o', 'Summarize this log', 500)}")
print(f"Actual: {round(actual_cost, 6)}")

Expected output shows two nearby floats. The actual prompt_tokens may be 5 instead of 4, nudging cost up by a fraction of a cent. Over millions of calls, that drift is real money.

Step 6: Account for cached input tokens

Providers increasingly honor cache-control hints to discount repeated prefixes. If your gateway forwards provider cache-control hints (as n4n.ai does), extend the estimator to separate cached tokens. Assume a 90% discount on cached input as an example—check your provider’s actual discount.

def estimate_with_cache(est: CostEstimator, model: str, input_text: str, output_tokens: int, cached_ratio: float = 0.0) -> float:
    price = est.pricing[model]
    in_tok = count_tokens(input_text, model)
    cached_tok = int(in_tok * cached_ratio)
    uncached_tok = in_tok - cached_tok
    cost = (uncached_tok / 1000) * price["input"]
    cost += (cached_tok / 1000) * price["input"] * 0.1
    cost += (output_tokens / 1000) * price["output"]
    return round(cost, 6)

If 70% of your system prompt is cached across requests, pass cached_ratio=0.7 and watch the input cost drop sharply. This matters when you build llm cost estimator budgets for agentic loops that replay the same instructions.

Step 7: Batch budgeting over a prompt corpus

A real app has thousands of distinct prompts. Load them from a JSONL file and aggregate.

import json as _json

def budget_corpus(est: CostEstimator, path: str, output_tokens: int, route: list[str], weights: list[float]) -> float:
    total = 0.0
    with open(path) as f:
        for line in f:
            row = _json.loads(line)
            text = row["prompt"]
            total += estimate_route(est, route, weights, text, output_tokens)
    return round(total, 4)

# Example prompts.jsonl: {"prompt": "Summarize this log"} per line
print(budget_corpus(est, "prompts.jsonl", 500, route, weights))

Checkpoint: with three lines of "Summarize this log", the printed total is roughly 3 * estimate_route(...) from Step 4. This is the number you hand to finance.

Closing notes

You now have a runnable pattern to build llm cost estimator logic that handles multiple models, fallback weighting, cache discounts, and real usage reconciliation. The hard part is not the math; it is keeping the pricing JSON current and piping actual usage records from your gateway into the same schema. Do that monthly and your forecasts will stay within a few percent of the bill.