Setting up per-model cost dashboards routing is the only way to keep LLM spend predictable when you fan out across multiple providers. Without a clear view of token cost per model per backend, you’re routing blind and will overpay for equivalent capability. This guide lays out a concrete path from raw usage logs to a dashboard that drives cheapest-provider routing in production.
1. Capture per-token usage at the edge
Route every LLM call through a single gateway or middleware that returns standardized usage. The OpenAI-compatible response shape includes a usage object; log it verbatim on each final response. With streaming, usage only appears on the last chunk, and some providers suppress it unless you pass stream_options: {include_usage: true}.
import json
import time
def log_completion(response: dict, provider: str, request_id: str):
usage = response.get("usage", {})
record = {
"ts": int(time.time()),
"provider": provider,
"model": response["model"],
"request_id": request_id,
"prompt_tokens": usage.get("prompt_tokens", 0),
"completion_tokens": usage.get("completion_tokens", 0),
"cached_tokens": usage.get("prompt_tokens_details", {}).get("cached_tokens", 0),
}
# ship to your sink: stdout, Kafka, Postgres
print(json.dumps(record))If you skip cached token counts, you will misattribute up to 90% discounts and your dashboards will lie. Most providers expose cache read pricing separately from fresh input tokens. Capture it from day one.
A common pitfall: trusting total_tokens without breaking out completion vs prompt. Completion tokens are typically 3–5x more expensive, and their ratio shifts with your prompt design. Keep them separate columns.
2. Normalize model identifiers and price sheets
Providers alias the same underlying model differently. gpt-4o-mini on one may be openai/gpt-4o-mini on another, and a gateway might prefix with its own slug. Build a static mapping and a price sheet keyed by canonical model ID. Treat the price sheet as code, not config: review in PRs.
{
"canonical_models": {
"4o-mini": ["openai/gpt-4o-mini", "azure/gpt-4o-mini", "n4n/gpt-4o-mini"],
"mixtral-8x7b": ["fireworks/mixtral-8x7b", "groq/mixtral-8x7b"]
},
"pricing": {
"openai/gpt-4o-mini": {"input_per_1k": 0.00015, "output_per_1k": 0.0006, "cache_read_per_1k": 0.000015},
"azure/gpt-4o-mini": {"input_per_1k": 0.00015, "output_per_1k": 0.0006, "cache_read_per_1k": 0.000015},
"groq/mixtral-8x7b": {"input_per_1k": 0.00012, "output_per_1k": 0.00012, "cache_read_per_1k": 0.00012}
}
}Flat per-1k pricing is a simplification. Many providers use tiered pricing (first 1M tokens cheaper). For a dashboard, flat averages over your observed mix are fine, but flag when a tier boundary is near. Also normalize all prices to USD; a provider billing in EUR will drift with exchange rates.
Model quality divergence is real. Two hosts of mixtral-8x7b are not identical after quantization and system prompt tweaks. Keep a manual override table so routing can exclude a provider from cheapest-provider selection per model.
3. Store usage in a queryable store
A narrow table beats a JSON blob when you need to aggregate. Postgres handles moderate scale (tens of millions of rows) with indexes on (model, ts). For high-volume inference, use ClickHouse with a MergeTree engine.
CREATE TABLE token_usage (
id BIGSERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL,
provider TEXT NOT NULL,
model TEXT NOT NULL,
request_id TEXT,
prompt_tokens INT NOT NULL,
completion_tokens INT NOT NULL,
cached_tokens INT NOT NULL
);
CREATE INDEX ON token_usage (model, ts);Backfill from gateway logs if you already run in production. Partition by week to keep inserts and deletes fast. Add a user_id or team_id column if you need per-team cost attribution—this is trivial to add now and painful later.
4. Build the per-model cost dashboards routing view
Compute effective cost per 1k tokens per provider using the price sheet joined to usage. This is the core of per-model cost dashboards routing: a single query that shows where each model is cheapest across backends.
WITH costs AS (
SELECT
provider,
model,
SUM(prompt_tokens - cached_tokens)/1000.0 * 0.00015 +
SUM(cached_tokens)/1000.0 * 0.000015 +
SUM(completion_tokens)/1000.0 * 0.0006 AS total_cost,
SUM(prompt_tokens + completion_tokens) AS total_tokens
FROM token_usage
WHERE ts > now() - interval '7 days'
GROUP BY provider, model
)
SELECT provider, model, total_cost, total_tokens,
total_cost / (total_tokens/1000.0) AS cost_per_1k
FROM costs
ORDER BY model, cost_per_1k;Pipe this into Grafana, Metabase, or a lightweight React table. Color the minimum cost per model row green. That visual is what your routing layer will consume. Use a 7-day window to absorb daily traffic spikes; a 24-hour view is too noisy for routing decisions.
If you prefer Python, the same aggregation in pandas:
import pandas as pd
df = pd.read_parquet("usage.parquet")
price = {"input":0.00015,"output":0.0006,"cache":0.000015}
df["cost"] = (
(df.prompt_tokens - df.cached_tokens)/1000*price["input"] +
df.cached_tokens/1000*price["cache"] +
df.completion_tokens/1000*price["output"]
)
agg = df.groupby(["model","provider"]).agg(total_cost=("cost","sum"),
total_tokens=("prompt_tokens","sum")).reset_index()
agg["cost_per_1k"] = agg.total_cost / (agg.total_tokens/1000)5. Implement cheapest-provider routing from dashboard data
Do not hardcode provider choice in app code. Generate a routing map from the dashboard query on a schedule (every 15 minutes is enough for stable price sheets). Apply latency and quality constraints before selecting the minimum.
type RouteRow = {model: string, provider: string, cost_per_1k: number, p95_ms: number};
type RouteMap = Record<string, string>; // canonical model -> provider
function buildRouteMap(rows: RouteRow[], max_latency_ms = 800): RouteMap {
const best: Record<string, {provider: string, cost: number}> = {};
for (const r of rows) {
if (r.p95_ms > max_latency_ms) continue; // skip slow backends
const canon = r.model.split('/').pop()!;
if (!best[canon] || r.cost_per_1k < best[canon].cost) {
best[canon] = {provider: r.provider, cost: r.cost_per_1k};
}
}
return Object.fromEntries(Object.entries(best).map(([k,v]) => [k, v.provider]));
}A gateway that honors client routing directives lets you pass provider preference per request. n4n.ai forwards such hints and falls back automatically when the chosen backend is rate-limited, so your dashboard-driven map stays safe under degradation. You send the same OpenAI-compatible request with an extra header or body field, and the gateway tries your preferred provider first.
Tradeoff: pure cheapest routing ignores latency variance. Constrain the map to providers whose p95 latency stays under a threshold you measure separately. Also weight cost by your actual traffic mix—a provider that is cheapest for 1% of calls shouldn’t displace a slightly pricier one serving 99%.
6. Common pitfalls and tradeoffs
Cache accounting. If you forward provider cache-control hints but fail to tag cached tokens in usage, your cost view skews high. Verify the cached_tokens field populates; some gateways report it under a different key.
Price sheet drift. Providers change prices without notice. A dashboard showing stale prices routes you to a provider that is no longer cheapest. Add a daily diff check against their published sheets and fail the build on mismatch.
Regional pricing and data residency. The cheapest provider may be in a region that violates your data residency rules or adds 200ms round trip. Filter by allowed regions before cost sorting.
Model quality divergence. Cheapest-provider routing must allow per-model opt-out when eval scores drop. Keep a manual override table reviewed by your ML team.
Fallback cost spike. Automatic fallback is good for reliability but can land on an expensive provider. Meter those falls and surface them as a separate line in the per-model cost dashboards routing view, or you’ll wonder why spend jumped 30% on a quiet Tuesday.
Rate limit headroom. A provider can be cheapest and permanently saturated. Track 429 rates per provider; exclude any above 1% from the route map.
7. Operationalize: refresh and alert
Schedule the route map build in cron or a CI job. Emit an alert when the cheapest provider for a high-traffic model changes, or when effective cost per 1k jumps >20% week over week.
*/15 * * * * /usr/local/bin/build_routemap.sh >> /var/log/routemap.log 2>&1Keep the last 30 days of dashboard data to spot seasonal patterns (e.g., end-of-month quota throttling). A simple z-score on daily cost_per_1k per model catches anomalies early:
import numpy as np
def alert_anomaly(series: np.ndarray, threshold=3.0):
z = (series[-1] - series.mean()) / (series.std() + 1e-9)
return abs(z) > thresholdPer-model cost dashboards routing is not a one-time project. It is a feedback loop: meter, visualize, route, measure again. The teams that do this systematically cut spend without sacrificing output quality or uptime.