When you route LLM traffic through automatic failover, cost anomalies fallback provider chains become invisible if you only look at aggregate spend. A request that starts on a cheap provider and silently retries to an expensive one blows your budget without changing the success rate. This guide shows how to instrument per-token metering and trace routing decisions so you can catch those anomalies before finance does.
Step 1: Emit structured usage records from every inference call
Your first job is to capture token counts and the serving provider on each response. Most OpenAI-compatible gateways return usage in the JSON body. If a request fails mid-retry, you may still incur input token costs on the attempted provider, so log the attempt even when the final status is an error. Failed attempts that processed the prompt before returning a 429 or 5xx often bill for those input tokens.
Wrap your client so nothing escapes unlogged:
import openai
from datetime import datetime, timezone
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def chat_with_logging(messages, request_id, model="auto"):
try:
resp = client.chat.completions.create(
model=model,
messages=messages,
extra_headers={"X-Request-Id": request_id}
)
usage = resp.usage
# provider field may be injected by gateway; fall back to model string
provider = resp.headers.get("X-Served-By", model)
log_usage(request_id, 0, provider, resp.model,
usage.prompt_tokens, usage.completion_tokens)
return resp
except openai.APIError as e:
# log the attempted provider from error headers if present
provider = e.headers.get("X-Attempted-Provider", "unknown")
log_usage(request_id, 0, provider, model, e.prompt_tokens or 0, 0)
raiseA gateway that provides per-token usage metering (such as n4n.ai) includes the final and attempted provider costs in the response headers, which removes guesswork. If you run your own retry loop, you must log each attempt manually and account for partial token consumption.
Step 2: Preserve routing lineage across retries
Cost anomalies fallback provider chains are defined by the sequence of providers a single logical request touches. A naive log per call loses the parent request. Propagate a stable request_id and collect each step.
If your gateway supports trace headers, read them:
# response header example from a fallback-aware gateway
trace = resp.headers.get("X-Routing-Trace") # "providerA:200,providerB:429,providerC:200"
for step_idx, hop in enumerate(trace.split(",")):
prov, status = hop.split(":")
log_attempt(request_id, step_idx, prov, status)When you control the retry loop, build the trace yourself:
def routed_chat(messages, request_id, providers):
trace = []
for i, prov in enumerate(providers):
try:
r = client.chat.completions.create(
model=f"route:{prov}",
messages=messages,
extra_headers={"X-Request-Id": request_id}
)
trace.append(f"{prov}:200")
log_usage(request_id, i, prov, r.model,
r.usage.prompt_tokens, r.usage.completion_tokens)
return r
except openai.RateLimitError:
trace.append(f"{prov}:429")
log_usage(request_id, i, prov, prov, 0, 0) # no tokens generated
raise RuntimeError("all providers failed")Keep the trace as a first-class column. You will group by it later. Note that some gateways forward provider cache-control hints; n4n.ai honors client routing directives and forwards those hints, so a cached prompt retry may show reduced input token cost on the fallback attempt. Capture that discount in your cost function.
Step 3: Store cost events with chain dimension
Write each attempt to a table that can be aggregated by (request_id, step_index). Use your contracted rates; do not hardcode public list prices because they drift and differ by commitment.
CREATE TABLE usage_events (
request_id TEXT,
step_index INT,
provider TEXT,
model TEXT,
prompt_tokens INT,
completion_tokens INT,
cost_cents NUMERIC(10,4),
ts TIMESTAMPTZ DEFAULT NOW()
);Compute cost_cents in application code:
def cost_for(provider, model, prompt_t, completion_t, cache_discount=0.0):
rate = get_contract_rate(provider, model) # returns (in_per_1k, out_per_1k)
in_rate, out_rate = rate
effective_in = prompt_t * (1 - cache_discount)
return (effective_in/1000)*in_rate*100 + (completion_t/1000)*out_rate*100Why the chain dimension matters
The chain for a request is the ordered list of provider values for its request_id. A cheap primary with an expensive fallback yields a different chain than a direct expensive call, and that distinction is the whole point. Aggregating only by final provider hides the overspend because the final provider looks normal.
Step 4: Compute expected vs actual cost per chain
Define the “expected” cost as what the primary provider would have charged had it succeeded. Actual cost is the sum of all attempts, including failed ones that may have processed prompts.
def chain_cost(request_id):
rows = query(f"SELECT provider, model, prompt_tokens, completion_tokens FROM usage_events WHERE request_id='{request_id}' ORDER BY step_index")
total = 0
for r in rows:
total += cost_for(r.provider, r.model, r.prompt_tokens, r.completion_tokens)
return total
def expected_cost(request_id, primary_provider):
first = query_first(f"SELECT model, prompt_tokens FROM usage_events WHERE request_id='{request_id}' AND step_index=0")
final = query_last(f"SELECT completion_tokens FROM usage_events WHERE request_id='{request_id}' ORDER BY step_index DESC LIMIT 1")
return cost_for(primary_provider, first.model, first.prompt_tokens, final.completion_tokens)A spike in actual - expected is the signal. The hardest part of tracking cost anomalies fallback provider chains is that the retry is invisible to naive logging, so this delta catches it. Run this calculation per request, then roll up by chain signature hourly.
Step 5: Detect anomalies with robust statistical bounds
Don’t alert on a static threshold; fallback rates fluctuate with provider health. Use a rolling median and median absolute deviation (MAD) per chain signature. Size the window to your traffic: 200 samples or 24 hours, whichever comes first.
import numpy as np
def detect_anomalies(chain_signature, window_costs, current_cost):
arr = np.array(window_costs)
med = np.median(arr)
mad = np.median(np.abs(arr - med)) or 1e-6
modified_z = 0.6745 * (current_cost - med) / mad
return modified_z > 3.5 # tunable sensitivityRun this per chain_signature (e.g., "provA->provB") every few minutes. If a normally cheap chain suddenly shows a 10x modified Z-score, page the on-call. The method resists single outliers from legitimate large prompts because it uses median, not mean.
Step 6: Alert and attribute to source
Attach the calling service and user to the request_id via metadata headers so finance can bill back. Forward the anomaly to your alerting pipe:
if detect_anomalies(sig, hist, cur):
slack.post({
"text": f"Cost anomaly on chain {sig}: expected {exp}c actual {act}c",
"tags": {"request_id": rid, "service": service, "owner": owner}
})Set alerts on cost anomalies fallback provider chains by comparing the delta per chain rather than per provider. This isolates the culprit: a misconfigured routing rule sending traffic to the wrong fallback tier, or a provider degradation forcing expensive retries during peak load.
Verify success
Deploy the logging wrapper and force a fallback in staging: point the primary provider at a blackhole rate limit and let the gateway retry. Within one aggregation window you should see a usage_events row for the failed attempt and a second for the fallback, with a positive actual - expected delta. The anomaly job should emit a Slack message tagged with the correct service.
If no message arrives, check that X-Routing-Trace (or your manual trace) is populated and that the cost function returns non-zero for the attempted provider. Send a test request with a known cache hint and confirm the cache_discount parameter reduces the logged cost. Once this pipeline runs for a week, review chains that never trigger alerts but still show consistent deltas—those are chronic silent overspends, not anomalies, and deserve a routing fix rather than a pager.