When you send LLM traffic through a gateway that fans out to multiple backends, a latency dashboard multi-provider routing view is the only way to see which provider is slow, which routing hint backfired, and where fallback kicked in. This tutorial builds a minimal metrics pipeline from raw API calls to a live Chart.js dashboard using Python and SQLite.
Prerequisites
- Python 3.10+ with
pip openaiandflaskinstalled (pip install openai flask)- An OpenAI-compatible endpoint URL and API key exported as
LLM_BASE_URLandLLM_API_KEY curlfor sanity checks
No frontend build tools required. We render charts with a CDN script.
Architecture
The design is deliberately boring:
- A Python client sends chat requests with optional routing headers.
- Client-side timing captures end-to-end latency including TLS and network.
- Each result is written to a local SQLite file.
- A Flask app serves the raw series and aggregated summaries as JSON.
- A static HTML page pulls those endpoints and draws line + bar charts.
This gives you a real latency dashboard multi-provider routing setup without a TSDB or hosted metrics agent.
Step 1: Measure latency and capture routing metadata
Provider-reported latency often excludes queue time. Measure on the client:
import time, openai, os
client = openai.OpenAI(
base_url=os.environ["LLM_BASE_URL"],
api_key=os.environ["LLM_API_KEY"],
)
def call_with_metrics(model, messages, routing_hint=None):
headers = {}
if routing_hint:
headers["X-Routing-Preference"] = routing_hint
start = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=messages,
extra_headers=headers,
stream=False,
)
elapsed_ms = (time.perf_counter() - start) * 1000
provider = resp.headers.get("x-provider", "unknown")
return {
"model": model,
"provider": provider,
"latency_ms": elapsed_ms,
"tokens": resp.usage.total_tokens,
"routing_hint": routing_hint,
}Expected output when printed:
{"model": "gpt-4o-mini", "provider": "openai", "latency_ms": 387.2, "tokens": 14, "routing_hint": null}If your gateway returns x-provider or similar, you get per-call provider attribution. If not, default to "unknown" and refine later.
Step 2: Persist to SQLite
Use the stdlib sqlite3 module. No ORM.
import sqlite3, time
conn = sqlite3.connect("latency.db")
conn.execute("""
CREATE TABLE IF NOT EXISTS calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts REAL,
model TEXT,
provider TEXT,
latency_ms REAL,
tokens INTEGER,
routing_hint TEXT
)
""")
def store(metrics):
conn.execute(
"INSERT INTO calls (ts, model, provider, latency_ms, tokens, routing_hint) VALUES (?,?,?,?,?,?)",
(time.time(), metrics["model"], metrics["provider"],
metrics["latency_ms"], metrics["tokens"], metrics["routing_hint"])
)
conn.commit()Run this once to create the schema. The file latency.db will appear in your working directory.
Step 3: Generate sample traffic
Loop over a few models and routing hints to populate data:
models = ["gpt-4o-mini", "claude-3-haiku", "mixtral-8x7b"]
hints = [None, "cost-optimized", "low-latency"]
for m in models:
for h in hints:
try:
metrics = call_with_metrics(m, [{"role": "user", "content": "ping"}], h)
store(metrics)
print("stored", metrics)
except Exception as e:
print("error", m, h, e)Checkpoint output:
stored {'model': 'gpt-4o-mini', 'provider': 'openai', 'latency_ms': 401.1, 'tokens': 12, 'routing_hint': None}
stored {'model': 'gpt-4o-mini', 'provider': 'openai', 'latency_ms': 388.7, 'tokens': 12, 'routing_hint': 'cost-optimized'}
...If a provider is rate-limited, the exception path prints error and you can see gaps in your series—exactly what the dashboard should reveal.
Step 4: Expose aggregated metrics via Flask
Spin up a tiny API alongside the collector:
from flask import Flask, jsonify, send_from_directory
app = Flask(__name__)
@app.route("/api/series")
def series():
cur = conn.cursor()
cur.execute("SELECT ts, provider, latency_ms FROM calls ORDER BY ts")
rows = cur.fetchall()
return jsonify([{"ts": r[0], "provider": r[1], "latency_ms": r[2]} for r in rows])
@app.route("/api/summary")
def summary():
cur = conn.cursor()
cur.execute("SELECT provider, AVG(latency_ms), COUNT(*) FROM calls GROUP BY provider")
return jsonify([{"provider": r[0], "avg_ms": r[1], "calls": r[2]} for r in cur.fetchall()])
@app.route("/")
def index():
return send_from_directory(".", "dashboard.html")Run with flask run --port 5000. Hit /api/summary with curl:
curl localhost:5000/api/summaryExpected:
[{"provider": "openai", "avg_ms": 395.4, "calls": 6}, {"provider": "anthropic", "avg_ms": 512.0, "calls": 6}]Step 5: Render the dashboard
Create dashboard.html:
<!doctype html>
<html>
<head>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
<script src="https://cdn.jsdelivr.net/npm/luxon@3/build/global/luxon.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chartjs-adapter-luxon@1"></script>
</head>
<body>
<canvas id="line" height="100"></canvas>
<canvas id="bar" height="100"></canvas>
<script>
async function load(){
const series = await fetch('/api/series').then(r=>r.json());
const summary = await fetch('/api/summary').then(r=>r.json());
const byProv = {};
series.forEach(p => {
(byProv[p.provider] = byProv[p.provider] || []).push({x: p.ts*1000, y: p.latency_ms});
});
new Chart(document.getElementById('line'), {
type: 'line',
data: { datasets: Object.entries(byProv).map(([p, pts]) => ({label: p, data: pts})) },
options: { scales: { x: { type: 'time' } } }
});
new Chart(document.getElementById('bar'), {
type: 'bar',
data: { labels: summary.map(s=>s.provider), datasets: [{label:'avg ms', data: summary.map(s=>s.avg_ms)}] }
});
}
load();
</script>
</body>
</html>Open http://localhost:5000. You should see a time-series line split by provider and a bar chart of average latency. That is your basic latency dashboard multi-provider routing view.
Step 6: Account for fallback and cache hints
Routing directives are useless if you can’t see when they were overridden. If your stack uses n4n.ai, which provides an OpenAI-compatible endpoint across 240+ models with automatic fallback, you can capture X-Fallback and X-Cache response headers to distinguish cached vs fresh provider latency. Extend the collector:
def call_with_metrics(model, messages, routing_hint=None):
headers = {}
if routing_hint:
headers["X-Routing-Preference"] = routing_hint
headers["X-Cache-Control"] = "max-age=3600" # forward cache hint
start = time.perf_counter()
resp = client.chat.completions.create(
model=model, messages=messages, extra_headers=headers, stream=False
)
elapsed_ms = (time.perf_counter() - start) * 1000
return {
"model": model,
"provider": resp.headers.get("x-provider", "unknown"),
"fallback": resp.headers.get("x-fallback", "false"),
"cache": resp.headers.get("x-cache", "miss"),
"latency_ms": elapsed_ms,
"tokens": resp.usage.total_tokens,
"routing_hint": routing_hint,
}Add fallback and cache columns to SQLite and filter them in the dashboard. A spike in fallback=true with high latency tells you the primary route is degraded—the whole point of building this.
Step 7: Run and verify end-to-end
Start the sampler in one shell:
python sampler.py # your script with Steps 1-3Start the server in another:
flask run --port 5000Then verify the series endpoint returns rows:
curl -s localhost:5000/api/series | head -c 200You should get JSON beginning with [{"ts": 1710000000.0, "provider": "openai", "latency_ms": 401.1}.
The dashboard is now live. To make it production-useful, schedule the sampler with cron or a worker loop, add a WHERE ts > now - 1h clause to the queries, and push the SQLite file to object storage for centralized viewing. The schema is small enough that you can keep months of per-call data locally without trouble.
A latency dashboard multi-provider routing setup like this takes less than an hour to stand up and immediately shows you whether your routing hints are saving money or silently pushing traffic to a slower backend.