Attributing LLM spend to features, not just API keys

Most teams wire up a single API key for their LLM provider and reconcile costs by service at the end of the month. That works until a PM asks why the “document summarizer” doubled spend week-over-week—attributing LLM spend to features demands tagging each call at the source, not reverse-engineering invoices.

Why API-key isolation falls short

Creating a separate key per feature seems obvious but collapses under reality: features call multiple models, models are shared across features, and key rotation becomes a deployment chore. You also lose the ability to track a single user journey that spans several features. Once you have more than five keys, audit trails and rotation policies eat engineering time that should go into product.

Step 1: Define a stable feature taxonomy

Before writing code, agree on a naming scheme. Use lowercase snake_case identifiers mapped to product surfaces:

• chat_compose
• rag_support_bot
• code_review_assist

Store this in a central config so services don’t freeform strings. High cardinality here will blow up your analytics later, so treat new feature IDs as a reviewed change. A feature should represent a distinct user-facing capability, not a code module; rag_support_bot is better than vector_query_helper.

Step 2: Propagate feature context on every request

OpenAI’s client doesn’t accept arbitrary metadata in the request body, but a gateway in front of the model can read headers. Set a default header on the client:

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.internal/v1",
    api_key="sk-...",
    default_headers={
        "X-Feature-Id": "rag_support_bot",
        "X-Env": "prod"
    }
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "How do I reset my password?"}]
)

If you must call the provider directly, the OpenAI SDK still lets you forward unknown fields via extra_body when your proxy reads them:

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hi"}],
    extra_body={"feature_id": "rag_support_bot"}
)

For raw HTTP, the header approach is cleanest:

curl https://gateway.internal/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Feature-Id: rag_support_bot" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hi"}]}'

Pitfall: never trust client-supplied headers in user-facing apps. Set the feature ID server-side based on the route or handler. A browser can send any X-Feature-Id; your gateway should overwrite it from authenticated context.

Step 3: Centralize metering at the edge

Logging token counts in each service duplicates logic and drifts when providers change usage fields. A gateway that already proxies your traffic can emit a structured usage event per request. For example, n4n.ai exposes per-token usage metering through one OpenAI-compatible endpoint covering 240+ models, so the feature header you sent is joined to billed tokens without custom instrumentation. That removes the need to parse responses in a dozen services.

Even without a managed gateway, you can middleware the response. For streaming calls, usage only appears in the final chunk:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Go"}],
    stream=True
)
usage = None
for chunk in stream:
    if chunk.usage:
        usage = chunk.usage
# usage.prompt_tokens / usage.completion_tokens now hold final counts

Emit those counts with the feature ID to a queue. Attributing LLM spend to features becomes a matter of subscribing to that queue rather than scraping provider dashboards.

Step 4: Store events with the right dimensions

Keep raw events in a columnar store or JSON logs. Minimal schema:

{
  "ts": "2024-05-12T10:22:31Z",
  "feature_id": "rag_support_bot",
  "model": "gpt-4o-mini",
  "prompt_tokens": 1200,
  "completion_tokens": 350,
  "provider": "openai",
  "cache_hit": false,
  "request_id": "req_8f2c"
}

Add provider and cache_hit because fallback routing and prompt caching change effective cost. Don’t precompute cost in the event unless you freeze a price table; store tokens and join pricing at query time. Version the price table so historical queries stay stable.

Step 5: Account for fallback and caching

Automatic fallback when a provider is rate-limited is a lifesaver, but your attribution must record the provider that actually served the token, not the one you requested. Forward the final provider name in the usage event. If you omit this, a degraded primary provider silently shifts traffic to a more expensive backup and your feature cost looks flat while the bill climbs.

Cache-control hints complicate token math. Anthropic and OpenAI discount cached input tokens; some gateways forward provider cache-control hints so you can flag cache_hit. OpenAI returns prompt_tokens_details.cached_tokens; Anthropic splits cache creation and cache read. If you miss this, you’ll overstate prompt token cost for repeated RAG contexts.

{
  "feature_id": "rag_support_bot",
  "model": "claude-3-5-sonnet",
  "prompt_tokens": 5000,
  "cache_creation_tokens": 4000,
  "cache_read_tokens": 1000,
  "completion_tokens": 200
}

I treat cache creation as a write cost attributed to the first call, and cache reads as near-zero. Pick a convention and document it.

Step 6: Query spend by feature

With events landed, a basic rollup shows where tokens go:

SELECT
  feature_id,
  model,
  SUM(prompt_tokens) AS prompt_tok,
  SUM(completion_tokens) AS completion_tok,
  SUM(prompt_tokens + completion_tokens) AS total_tok
FROM usage_events
WHERE ts >= '2024-05-01' AND ts < '2024-06-01'
GROUP BY feature_id, model
ORDER BY total_tok DESC;

Join a price table to get dollars. Keep the price table versioned; model prices change and you don’t want to retroactively rewrite history. For cached tokens, adjust the join to apply the discounted rate.

Step 7: Close the loop with product

Export the daily feature cost to a dashboard PMs actually open. When chat_compose spikes, you can see if it’s because of longer system prompts or a new user flow. This turns attributing LLM spend to features from a finance exercise into an engineering lever—you can cap system prompt size or switch models for a specific feature without affecting others.

Common pitfalls

• Header spoofing: If the gateway is public, validate feature IDs against an allowlist.
• Streaming blind spots: Usage only appears in the last chunk. If you log on request start, you’ll record zero tokens.
• Model drift: A feature silently switched to a larger model via config; your taxonomy won’t catch it unless you group by model too.
• Missing fallback dimension: Counting everything as the primary provider hides cost spikes from degraded upstreams.
• Over-aggregation: Rolling up to daily totals too early makes it impossible to debug a single anomalous request.

Tradeoffs

Adding attribution headers is cheap but forces you to own a taxonomy. Centralizing at a gateway reduces code but introduces a dependency. Storing raw token events preserves flexibility at the cost of storage volume. For most teams, the taxonomy plus gateway metering hits the sweet spot before you exceed a few million requests per month. Beyond that, partitioning the events by feature and date keeps query costs bounded.