Monitoring token-per-second throughput across providers

Token-per-second throughput monitoring is the only reliable way to compare LLM inference performance across providers, because a single latency number hides whether the model is streaming slowly or just queuing before first byte. Without measuring generation speed at the token level, you cannot tell if a 12-second response came from a cold GPU or a throttled API. This guide lays out an ordered path to instrument, normalize, and alert on token-per-second throughput monitoring across heterogeneous backends so your dashboards reflect reality, not averages distorted by network hops.

1. Capture raw stream timings at the client

Start by measuring the two phases that matter: time to first token (TTFT) and generation interval. Most OpenAI-compatible endpoints support streaming with stream_options={"include_usage": True}, which emits a final chunk containing completion_tokens. Use that for ground-truth counts rather than estimating from character length.

import time
from openai import OpenAI

client = OpenAI()  # point base_url at your gateway or provider

start_req = time.perf_counter()
first_token_ts = None
last_token_ts = None
usage = None

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize the CAP theorem."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.usage:
        usage = chunk.usage
    if chunk.choices and chunk.choices[0].delta.content:
        now = time.perf_counter()
        if first_token_ts is None:
            first_token_ts = now
        last_token_ts = now

ttft = first_token_ts - start_req
gen_duration = last_token_ts - first_token_ts
tps = usage.completion_tokens / gen_duration if gen_duration > 0 else 0

The tps value is your native token-per-second throughput monitoring sample. Do not include TTFT in the denominator; queueing and prefill are separate failure modes. If you run concurrent requests, wrap this in an async loop and avoid blocking the event loop with token counting.

Pitfall: client-side buffering

Some SDKs or proxies buffer chunks. If you see exactly one chunk containing the full text, your first_token_ts equals last_token_ts and TPS explodes. Verify by logging chunk counts; a healthy stream yields dozens to hundreds of deltas. Test with curl --no-buffer against the raw endpoint to confirm chunked transfer encoding is intact.

2. Normalize token counts across tokenizers

A token from Claude is not the same size as a token from GPT-4. Raw TPS comparisons across providers are apples-to-oranges. For token-per-second throughput monitoring that spans vendors, maintain two metrics: native TPS (using provider usage) and normalized text throughput (bytes or Unicode chars per second).

char_count = 0
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        char_count += len(chunk.choices[0].delta.content)
char_throughput = char_count / gen_duration

Normalized throughput loses tokenization detail but reveals whether one provider ships denser text. Use it as a secondary signal, not for capacity planning. Expect roughly 3–5 characters per token on English text, but do not hardcode that ratio—non-Latin scripts deviate sharply.

3. Tag every sample with routing context

Push each measurement to your metrics system with labels for provider, model, region, and whether the request hit a fallback path. Prometheus example:

from prometheus_client import Histogram

TPS_HIST = Histogram(
    "llm_gen_tokens_per_second",
    "Generation throughput",
    ["provider", "model", "fallback"],
)

TPS_HIST.labels(provider="openai", model="gpt-4o-mini", fallback="false").observe(tps)

If you call multiple providers directly, set the label from your own routing logic. If you sit behind a gateway, read the served-provider metadata from response headers or the usage object. n4n.ai, for instance, returns per-token usage metering and honors client routing directives, so the actual backend that processed the request is unambiguous—critical for clean token-per-second throughput monitoring when automatic fallback kicks in. Without that provenance, a silent reroute contaminates your baseline.

4. Separate TTFT and generation TPS on the dashboard

A dashboard that plots only median TPS will mask a provider that takes 8 seconds to start streaming then runs fast. Build two panels:

• TTFT histogram by provider/model.
• Generation TPS line with rolling 5-minute average.

Tradeoff: high generation TPS with terrible TTFT yields a sluggish feel in chat UIs. Weigh them by product need; a batch summarizer cares about TPS, an interactive agent cares about TTFT. If you must combine into one SLO, weight TTFT as 40% of perceived latency based on Jakob Nielsen’s tolerance research, but keep the raw series available.

5. Alert on relative regression, not static floors

Static alerts like “TPS < 20” break when you swap models. Instead, compute a baseline per label over the prior 24 hours and alert when the 50th percentile drops >25% for 10 minutes.

# Prometheus alerting rule (simplified)
- alert: ThroughputRegression
  expr: |
    histogram_quantile(0.5, rate(llm_gen_tokens_per_second_bucket[10m]))
    < 0.75 * histogram_quantile(0.5, rate(llm_gen_tokens_per_second_bucket[24h]))
  for: 10m
  labels: {severity: warning}

This catches provider degradation, regional throttling, or a bad fallback route without manual threshold tuning. For high-traffic systems, add a multi-window alert (short 5m vs long 1h) to reduce flapping.

6. Handle fallback and mixed-model traffic

When a gateway fails over from a primary to a secondary provider, the model may change (e.g., GPT-4o to a smaller fallback). Your token-per-second throughput monitoring must either split by served model or exclude fallback samples from baseline. Otherwise a correct fallback event looks like a 3x throughput improvement and silences real alerts.

Implement a fallback label as shown in step 3, and filter it out of baseline queries unless you explicitly want to track fallback performance. If your gateway supports cache-control hints, note that a cache hit should be recorded as cache="true" and excluded from backend TPS entirely—it is not inference.

Tradeoff: metering overhead

Calling a tokenizer client-side to count tokens defeats the purpose—you add CPU latency and still mismatch provider vocab. Always prefer the provider’s usage field. If a provider doesn’t supply streaming usage, fall back to normalized char throughput and mark the sample token_count="estimated".

7. Common pitfalls in production

• Clock resolution: time.time() on some containers has millisecond jitter. Use time.perf_counter().
• Proxy buffering: NGINX or CDN may buffer streaming responses. Test with curl --no-buffer to confirm chunked delivery.
• Concurrent load: Measuring TPS from a single serial client misses saturation behavior. Run distributed workers and aggregate.
• Ignoring prefill: Long prompts inflate TTFT. Track prompt token count alongside TTFT to distinguish prefill cost from queue delay.
• TLS handshake: A cold connection adds 100–300ms. Reuse HTTP sessions or measure from first request byte sent, not socket open.
• HTTP/2 multiplexing: Shared connections can cause head-of-line blocking; pin streams to separate connections if TPS variance is high.

8. Store raw samples for post-hoc analysis

Aggregated histograms lose tail behavior. Ship raw samples (timestamp, provider, model, tps, ttft, prompt_tokens) to a columnar store like ClickHouse or BigQuery. This lets you answer “what was TPS for requests between 2–4k prompt tokens to provider X on Tuesday?” after an incident. Keep raw retention at 7 days; roll up to hourly stats beyond that.

9. Minimum viable implementation checklist

1. Stream with include_usage and record first_token_ts, last_token_ts.
2. Compute native TPS and char throughput; emit both.
3. Label with provider, model, fallback, region, cache.
4. Dashboard TTFT and TPS separately.
5. Alert on 25% relative drop vs 24h baseline.
6. Filter fallback and cache hits from primary baselines.
7. Retain raw samples for at least a week.

Following this path gives you defensible token-per-second throughput monitoring that survives provider changes, routing tweaks, and traffic spikes. The metric stops being a vanity number and becomes the early warning system your LLM stack actually needs.