A cost optimization checklist for LLM API usage

Shipping LLM features without a disciplined LLM cost optimization checklist is how teams blow through their inference budget in a week. This post lays out the concrete steps we use to keep token spend predictable while maintaining output quality, from model selection to request-level routing.

1. Right-size the model to the task

The first item on any LLM cost optimization checklist is matching model capability to the actual requirement. Calling a 70B-class frontier model to classify “positive/negative” sentiment or extract a date from text is pure waste. Use 7B–13B instruction-tuned models for extraction, routing, and structured output; reserve large models for open-ended reasoning where smaller ones clearly fail.

Build a tiered model registry so swaps are config, not code:

MODEL_TIERS = {
    "classify": "mistral-7b-instruct",
    "extract": "gpt-4o-mini",
    "reason": "gpt-4o",
}

def complete(task: str, prompt: str):
    model = MODEL_TIERS[task]
    return client.chat.completions.create(model=model, messages=[{"role": "user", "content": prompt}])

Expect 5–10x cost reduction on auxiliary tasks by dropping a tier. For deterministic jobs (JSON extraction, PII redaction), small models often hit 98% of the large model’s accuracy at 1/8 the cost. Log eval scores alongside spend instead of assuming smaller means worse.

2. Enforce token budgets at the request level

Unbounded max_tokens and giant system prompts are silent budget killers. A solid LLM cost optimization checklist always includes enforcing token budgets at the request level: set explicit output caps and trim conversation history before sending.

def trimmed_messages(history, max_context_tokens=2000):
    while estimate_tokens(history) > max_context_tokens:
        history.pop(0)
    return history

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=trimmed_messages(history),
    max_tokens=256,  # hard cap on generation
)

If your gateway supports it, forward provider cache-control hints so repeated prefixes aren’t re-billed. n4n.ai forwards provider cache-control hints on its OpenAI-compatible endpoint, which means your long system prompt can be cached without custom integration.

3. Cache prompts and completions aggressively

Every identical prompt re-sent to the API is pure waste. Implement a semantic or exact-match cache keyed on normalized prompt + model + params. For deterministic tasks, a Redis cache cuts repeat calls to zero.

import hashlib, json, redis

r = redis.Redis()

def cached_complete(model, messages, **kwargs):
    key = hashlib.sha256(json.dumps([model, messages, kwargs]).encode()).hexdigest()
    if (hit := r.get(key)):
        return json.loads(hit)
    resp = client.chat.completions.create(model=model, messages=messages, **kwargs)
    r.setex(key, 3600, json.dumps(resp.model_dump()))
    return resp

For longer contexts, use provider prompt caching. Send the cache breakpoint explicitly:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"claude-3-5-sonnet","messages":[{"role":"system","content":"...long static...","cache_control":{"type":"ephemeral"}}]}'

The gateway passes that hint through, so you pay write cost once.

4. Batch and parallelize independent calls

If you need summaries for 1000 documents, don’t loop serially. Use batch APIs or async concurrency to saturate rate limits and reduce retry overhead.

import asyncio

async def summarize(doc):
    return await client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":doc}])

async def batch(docs):
    return await asyncio.gather(*[summarize(d) for d in docs])

Many gateways offer asynchronous batch endpoints with 24h turnaround at 50% cost. If latency isn’t critical, that’s an easy win with zero quality impact.

5. Route to cheaper providers with fallback

A single provider outage or rate limit shouldn’t force you to the most expensive model. Encode routing preferences in the request and let a gateway handle degradation. For example, prefer a cheap provider but allow fallback:

{
  "model": "auto",
  "messages": [{"role": "user", "content": "Translate this."}],
  "route": {
    "prefer": ["groq/llama-3-70b", "together/qwen-72b"],
    "fallback": ["openai/gpt-4o-mini"]
  }
}

An OpenRouter-class gateway like n4n.ai honors those client routing directives and performs automatic fallback when a provider is rate-limited or degraded, so you keep spend near your preferred tier without writing retry logic.

Don’t forget per-token metering

Attach a user/feature tag to each call. Per-token usage metering lets you attribute cost to teams or endpoints and spot leaks. The gateway’s usage response should include usage.total_tokens and a computed cost field.

6. Meter usage per token and alert on anomalies

You can’t optimize what you don’t measure. Capture usage.prompt_tokens and usage.completion_tokens on every response, push to metrics, and alert when a single trace exceeds baseline by 2x.

@app.middleware("http")
async def log_usage(request, call_next):
    resp = await call_next(request)
    if hasattr(resp, "usage"):
        statsd.incr("llm.tokens", resp.usage.total_tokens, tags=[f"model:{resp.model}"])
    return resp

Set weekly budgets per service. When the meter shows a spike from a new prompt template, you fix it before finance does.

7. Use streaming to cancel wasted generation

If your app shows partial results or has a validation step, stream tokens and abort the request when output is clearly off-track. This avoids paying for 2k generated tokens that you discard.

stream = client.chat.completions.create(model="gpt-4o", messages=msgs, stream=True)
for chunk in stream:
    if bad_token(chunk):
        stream.close()  # cancels billing for remaining tokens on most providers
        break

Not all providers support zero-cost cancellation, but most stop metering at disconnect.

8. Prune API keys, endpoints, and dead experiments

Audit your key inventory quarterly. A forgotten staging key with no budget cap is a classic horror story. Disable unused model endpoints and rotate keys with scoped permissions.

curl -H "Authorization: Bearer $ADMIN" https://api.n4n.ai/v1/keys
curl -X DELETE -H "Authorization: Bearer $ADMIN" https://api.n4n.ai/v1/keys/key_123

Centralize access through one gateway with per-token metering so offboarding is one call, not ten.

Summary

Run through this LLM cost optimization checklist before every new LLM feature ships. The steps are boring, but they are the difference between a demo that costs $5 and a production bill that costs $50k.