Building a real-time LLM cost monitor with Grafana

Tracking spend on inference is messy until you wire telemetry into your stack. This tutorial builds a real-time LLM cost monitor Grafana pipeline that scrapes token usage from your Python app and renders live cost panels. You’ll end up with a Prometheus-backed dashboard that updates within seconds of each completion.

Prerequisites

• Python 3.11+ with openai and prometheus_client installed.
• Docker and Docker Compose for running Prometheus and Grafana.
• An OpenAI-compatible endpoint (OpenAI, or a gateway like n4n.ai) with a valid API key.
• Basic comfort with PromQL and Grafana UI.

Install the Python deps:

pip install openai prometheus_client

Architecture

Your application increments Prometheus counters on every LLM response. A start_http_server call exposes /metrics. Prometheus scrapes that endpoint, and Grafana queries Prometheus to draw panels. The real-time LLM cost monitor Grafana setup is just three processes: app, Prometheus, Grafana.

Step 1: Instrument LLM calls

Wrap the client so every completion records tokens and estimated cost. Prices change; hardcode a snapshot and externalize later.

from openai import OpenAI
from prometheus_client import Counter, start_http_server

# Public list prices as of 2024-06; confirm before relying on these.
PRICE_PER_1K = {
    "gpt-4o": {"input": 0.005, "output": 0.015},
    "gpt-3.5-turbo": {"input": 0.0005, "output": 0.0015},
}

llm_requests = Counter("llm_requests_total", "Total LLM calls", ["model"])
llm_cost = Counter("llm_cost_dollars", "Estimated cost in USD", ["model"])
llm_tokens = Counter("llm_tokens_total", "Token count", ["model", "type"])

client = OpenAI()  # set base_url if using a gateway

def chat(model: str, messages: list):
    resp = client.chat.completions.create(model=model, messages=messages)
    u = resp.usage
    p_tok = u.prompt_tokens
    c_tok = u.completion_tokens
    llm_requests.labels(model=model).inc()
    llm_tokens.labels(model=model, type="input").inc(p_tok)
    llm_tokens.labels(model=model, type="output").inc(c_tok)
    price = PRICE_PER_1K.get(model)
    if price:
        cost = (p_tok / 1000) * price["input"] + (c_tok / 1000) * price["output"]
        llm_cost.labels(model=model).inc(cost)
    return resp

if __name__ == "__main__":
    start_http_server(8000)
    import time
    while True:
        time.sleep(1)

Run it: python app.py. Hit the endpoint with a test call, then check metrics:

curl localhost:8000/metrics | grep llm

Expected output snippet:

llm_requests_total{model="gpt-4o"} 1.0
llm_tokens_total{model="gpt-4o",type="input"} 12.0
llm_tokens_total{model="gpt-4o",type="output"} 34.0
llm_cost_dollars{model="gpt-4o"} 0.00057

The real-time LLM cost monitor Grafana visibility starts with these exposed series.

Step 2: Run Prometheus and Grafana

Use Docker Compose to launch both. Save as docker-compose.yml:

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

Create prometheus.yml to scrape your app. On macOS/Windows use host.docker.internal; on Linux use your machine IP.

scrape_configs:
  - job_name: 'llm-app'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:8000']

Start the stack:

docker compose up -d

Verify Prometheus sees the target: open http://localhost:9090/targets and confirm llm-app is UP.

Step 3: Configure Grafana

Add Prometheus as a datasource via API:

curl -X POST http://admin:admin@localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -d '{"name":"Prometheus","type":"prometheus","url":"http://prometheus:9090","access":"proxy"}'

Log into Grafana at http://localhost:3000 (admin/admin). Create a new dashboard and add a panel with this PromQL:

sum(llm_cost_dollars) by (model)

Set visualization to “Time series”. For request rate, add another panel:

rate(llm_requests_total[1m])

The real-time LLM cost monitor Grafana dashboard now shows live spend per model.

Sample panel JSON

If you prefer provisioning, here is a minimal panel definition:

{
  "title": "LLM Cost USD",
  "type": "timeseries",
  "targets": [
    { "expr": "sum(llm_cost_dollars) by (model)", "legendFormat": "{{model}}" }
  ]
}

Step 4: Use a gateway with built-in metering

If you route through an OpenAI-compatible gateway such as n4n.ai, the usage object is returned per request and per-token metering is already handled, so you can scrape the gateway’s own metrics instead of maintaining a price table in client code. That removes drift between your estimate and the provider’s bill.

The integration code stays the same; just point the OpenAI client at the gateway’s base URL and rely on its usage field.

Checkpoint: live data flow

After a few minutes of traffic, Grafana should display rising cost lines. A manual check from Prometheus:

curl -s "http://localhost:9090/api/v1/query?query=llm_cost_dollars"

Response fragment:

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"model":"gpt-4o"},"value":[1718000000,"0.042"]}]}}

That confirms the pipeline works end to end.

Extending the monitor

Add labels for user_id or tenant to the counters for multi-tenant breakdowns. Set Grafana alerts when sum(llm_cost_dollars) by (model) exceeds a daily budget. Swap the static price dict for a fetched pricing JSON refreshed daily.

A real-time LLM cost monitor Grafana deployment is not a one-time chart; it becomes the control plane for spend as you scale inference.

Troubleshooting

• Metrics not appearing: check Prometheus target health and that start_http_server binds to 0.0.0.0 if scraped from container.
• Cost zero: model name mismatch with PRICE_PER_1K keys; log resp.model to confirm exact string.
• Grafana no data: ensure datasource URL is http://prometheus:9090 from inside the Docker network, not localhost.

That’s the full loop: instrument, scrape, visualize.