Dashboards for tracking prompt vs completion token costs

Most teams watch LLM spend as a single monthly total until a spike forces a postmortem. Building prompt vs completion token cost dashboards turns that opaque number into actionable signals about context size, model verbosity, and caching efficiency.

Why the split matters

Prompt tokens and completion tokens are priced differently by every provider. In many models, completion tokens cost 2–4x more per unit than prompt tokens. If you only track total tokens, you cannot tell whether your bill grew because you stuffed more context into the system prompt or because the model started writing novels. Separating them lets you target the right fix: trim retrieved documents, constrain output length, or switch models.

The split also exposes caching ROI. Providers discount cached prompt tokens heavily. A dashboard that lumps them into full-price prompt tokens hides the savings you already earned.

Step 1: Emit structured usage events

Capture token counts at the point of inference. Every OpenAI-compatible API returns a usage object with prompt_tokens and completion_tokens. Log these alongside the model name returned by the response, timestamp, and cache metadata. Do not trust the model you requested; under fallback the served model differs.

import time, json
from openai import OpenAI

client = OpenAI(base_url="https://api.your-gateway.com/v1", api_key="sk-...")

def complete(messages, model):
    resp = client.chat.completions.create(model=model, messages=messages)
    u = resp.usage
    cached = 0
    if hasattr(u, "prompt_tokens_details"):
        cached = u.prompt_tokens_details.cached_tokens or 0
    event = {
        "ts": int(time.time()),
        "model": resp.model,          # use served model, not requested
        "prompt_tokens": u.prompt_tokens,
        "completion_tokens": u.completion_tokens,
        "cached_prompt_tokens": cached,
        "request_id": resp.id,
    }
    # ship to Kafka, PubSub, or stdout in dev
    print(json.dumps(event))
    return resp

Emit raw events; let a downstream job handle pricing and grouping. This keeps instrumentation decoupled from price changes. If you log inside a request path, push to a queue instead of blocking on a database write.

Step 2: Store events with immutable model metadata

Write events to a table or columnar store. Keep raw token counts separate from price. Maintain a second table model_prices keyed by model and effective date, because providers change prices and introduce tiers.

CREATE TABLE usage_events (
    ts BIGINT,
    model TEXT,
    prompt_tokens INT,
    completion_tokens INT,
    cached_prompt_tokens INT,
    request_id TEXT
);

CREATE TABLE model_prices (
    model TEXT,
    effective_date DATE,
    prompt_price_per_1k NUMERIC,
    completion_price_per_1k NUMERIC,
    cache_read_price_per_1k NUMERIC
);

Partition usage_events by day for query speed. Tradeoff: storing per-request rows scales linearly with traffic. For high volume, pre-aggregate to minute buckets after a 30-day raw retention window. Keep raw long enough to reconcile invoices.

Step 3: Build the prompt vs completion token cost dashboards

Now the actual prompt vs completion token cost dashboards. Query the joined data, grouping by day and model.

SELECT
    date_trunc('day', to_timestamp(ts)) AS day,
    e.model,
    SUM(prompt_tokens) AS prompt_tok,
    SUM(completion_tokens) AS completion_tok,
    SUM(cached_prompt_tokens) AS cached_tok,
    SUM(prompt_tokens * p.prompt_price_per_1k / 1000.0) AS prompt_cost,
    SUM(completion_tokens * p.completion_price_per_1k / 1000.0) AS completion_cost,
    SUM(cached_prompt_tokens * p.cache_read_price_per_1k / 1000.0) AS cache_cost
FROM usage_events e
JOIN model_prices p ON e.model = p.model
WHERE p.effective_date <= to_timestamp(e.ts)::date
GROUP BY day, e.model
ORDER BY day DESC;

Choose aggregation granularity

Hourly granularity catches regressions from a bad deploy. Daily is enough for finance. Expose both via a dropdown in your BI tool (Grafana, Metabase, or Looker). Avoid plotting raw per-request points past a few thousand rows; pre-aggregate.

Plot ratio, not just absolute

A stacked bar of prompt vs completion cost is obvious. Add a line for completion_cost / (prompt_cost + completion_cost + cache_cost) to spot drift toward verbose outputs. A rising ratio means your prompts are stable but the model talks more—fix with max_tokens or a stricter system prompt.

Step 4: Account for fallback and cache hints

If you route through a gateway that fails over to a secondary provider, your dashboard must attribute tokens to the model that actually served the request, not the one you requested. n4n.ai meters per-token usage and forwards provider cache-control hints, so captured cached_prompt_tokens reflect reality instead of your assumption. Without this, you undercount savings from prompt caching and misattribute cost when a fallback model has different pricing. Always log the model field from the response object.

Common pitfalls and tradeoffs

• Ignoring cached tokens. Many providers discount cached prompt tokens. If you lump them into full-price prompt tokens, you overstate cost and miss the ROI of caching.
• Using total_tokens only. Total tokens hide the split. Always persist the two components separately.
• Hardcoding prices in the app. Prices change. A lookup table with effective dates prevents dashboard rewrites.
• Mixing model families in one series. GPT-4o and Mistral-7B have different cost structures; group by model or tag.
• Real-time vs batch. Streaming events to a warehouse gives fresh dashboards but adds pipeline cost. Batch every 5 minutes is usually enough for engineering use.
• Forgetting currency and tier. Some providers price by region or volume tier. Include tier in model_prices if applicable.

A minimal reference dashboard in Python

If you lack a BI tool, a static script with matplotlib can validate the pipeline.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_sql("""
    SELECT day, model,
           SUM(prompt_cost) AS pc,
           SUM(completion_cost) AS cc
    FROM dashboard_view GROUP BY day, model
""", conn)

pivot = df.pivot(index='day', columns='model', values=['pc','cc'])
pivot.plot(kind='bar', stacked=True)
plt.ylabel("USD")
plt.title("Prompt vs completion token cost dashboards by model")
plt.show()

This is not a replacement for a real dashboard, but it confirms your joins work and prices are applied.

Operationalize alerts

Once prompt vs completion token cost dashboards are live, alert on anomalies: completion cost per request exceeding a threshold, or cache hit rate dropping below 50%. Those signals precede invoice surprises. Set the alert on the ratio, not absolute spend, because traffic grows naturally. A sudden jump in completion proportion is a code or prompt regression.

Wire alerts to Slack or PagerDuty with a link to the dashboard filtered to the offending model and hour.

Closing checklist

• Emit raw token events with response model and cache details.
• Store prices in a dated table; never hardcode.
• Build dashboards that separate prompt, completion, and cached costs.
• Validate with the actual served model, especially under fallback.
• Alert on ratio shifts, not just total spend.

That is the path from opaque API bill to controllable unit economics.