Tokens-per-minute limits and their effect on real throughput

Tokens per minute limit throughput is the metric most teams cite when sizing LLM capacity, but it obscures more than it reveals. The nominal TPM cap published by an API provider describes a billing and safety boundary, not the rate at which your application will actually process tokens under concurrent load. If you size your infrastructure against the printed number, you will either over-provision cost or starve your users during traffic spikes.

Why providers impose tokens-per-minute caps

API gateways enforce a tokens per minute limit to protect shared GPU pools from a single tenant. Transformer inference consumes VRAM proportionally to sequence length, and batch scheduling works best when request volume is smooth. A rolling window counter that sums prompt and completion tokens across all calls lets the provider shed load predictably.

The limit is an average, not a bandwidth guarantee. A 200,000 TPM quota means you can send 200k tokens in a 60-second sliding window; it does not mean the model will generate them in that minute. The actual generation speed is bounded by model architecture, batch size, and hardware. A 70B-class model on A100 might sustain 30–50 tokens per second per sequence; a smaller distilled model can exceed 150 tokens per second. Those are physical rates that no quota relaxes.

The gap between nominal cap and realized throughput

Consider a support chatbot. Each interaction sends a 1,200-token system prompt plus user message and receives ~300 completion tokens. Naively, a 100k TPM limit supports 66 conversations per minute (100,000 / 1,500). That arithmetic ignores latency.

If the model takes 4 seconds to return the 300 tokens, a single synchronous client achieves at most 15 requests per minute, burning only 22.5k tokens of the quota. To approach the TPM ceiling you must run many requests in parallel. But providers couple TPM with requests-per-minute (RPM) and concurrent connection limits. Exceed RPM and you get 429s; exceed concurrency and requests queue in your own client.

Worse, throttling is not instantaneous. When you hit the window limit, the API returns 429 with a retry-after hint. Your client backs off, leaving the quota partially unused. The effective tokens per minute limit throughput becomes a function of your retry strategy and traffic shape, not the printed number.

A minimal load test

Measure instead of guess. Below is an asyncio script that hammers an OpenAI-compatible endpoint and records actual completed tokens per minute.

import asyncio, time, os
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url=os.environ["LLM_BASE"], api_key="dummy")

async def one_request():
    resp = await client.chat.completions.create(
        model="mistral-7b-instruct",
        messages=[{"role":"user","content":"Summarize: " + "x"*1000}],
        max_tokens=200,
    )
    return resp.usage.total_tokens

async def main(concurrency: int, duration: float):
    start = time.monotonic()
    completed_tokens = 0
    async def worker():
        nonlocal completed_tokens
        while time.monotonic() - start < duration:
            try:
                completed_tokens += await one_request()
            except Exception as e:
                await asyncio.sleep(1)  # crude backoff
    tasks = [asyncio.create_task(worker()) for _ in range(concurrency)]
    await asyncio.gather(*tasks)
    elapsed = time.monotonic() - start
    print(f"Concurrency={concurrency} -> {completed_tokens/elapsed:.0f} tok/s | {completed_tokens/elapsed*60:.0f} effective TPM")

for c in [4, 16, 32]:
    asyncio.run(main(c, 60))

Run this against your target model. You will typically see a knee: low concurrency underutilizes the quota, moderate concurrency saturates it, and high concurrency triggers 429 storms that drop effective throughput below the knee.

What actually drives tokens per minute limit throughput

Three variables dominate:

1. Generation latency – Time to first token plus inter-token delay. Streaming does not change total token count but improves perceived latency; it does not raise TPM.
2. Prompt overhead – Every token you send counts. A 2k-token RAG context that yields 50 tokens of answer consumes 97% of your quota on input. Cache prompts where the provider supports it.
3. Concurrency ceiling – You need enough in-flight requests to keep the model busy, but not so many that you trip RPM or connection limits.

A gateway that honors client routing directives and forwards provider cache-control hints can recover lost throughput. For example, n4n.ai forwards cache-control headers so repeated system prompts hit provider prompt caches, reducing billed and counted input tokens on subsequent calls. That effectively raises your real tokens per minute limit throughput without negotiating a higher quota.

Rolling window mechanics matter

Not all TPM limits are implemented as a fixed 60-second bucket. Many providers use a sliding window that continuously admits requests as old ones expire. This permits short bursts above the average rate, followed by a period where you can send almost nothing. If your traffic is spiky, the sliding window will clip your peaks harder than a fixed window would.

Inspect the headers on a real response to see your state:

curl -i https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"test","messages":[{"role":"user","content":"hi"}]}'
# Watch for: x-ratelimit-remaining-tokens, x-ratelimit-reset-tokens, retry-after

If x-ratelimit-remaining-tokens hits zero mid-burst, your next requests will 429 regardless of how low your average is. Designing for the average tokens per minute limit throughput without accounting for window shape is a mistake.

Batching and request shaping tradeoffs

You can reshape traffic to squeeze more out of a fixed TPM. Combine multiple independent queries into one completion with delimiters, then split client-side. This cuts per-request overhead and RPM pressure, but it increases prompt tokens and risks hitting max sequence length. For a 100k TPM limit, shipping 10 questions in one 3k-token prompt returning 500 tokens uses 3.5k tokens per batch versus 10×(1k+50)=10.5k if sent separately. The batch wins on quota efficiency but loses on latency isolation—one slow sub-task stalls the whole batch.

Streaming with stream: true lets you start processing tokens early, but the total tokens counted against the limit are identical. Do not mistake time-to-first-byte improvements for throughput gains.

The cost of ignoring the gap

Teams that trust the nominal cap build brittle systems. They set their load balancer to allow exactly the quoted TPM, then watch p99 latency climb as soon as traffic patterns shift to longer prompts. Or they buy a higher tier expecting 2× throughput, only to find their client is single-threaded and still bounded by generation speed.

I have seen a production outage caused by a 24k-token system prompt rolled out to 500 concurrent sessions. The provider’s 300k TPM limit looked ample on paper: 300k / 24k = 12.5 sessions per minute. They flooded the API with 500 parallel requests, got mass 429s, and their retry loop amplified the load. Effective tokens per minute limit throughput dropped to near zero for five minutes.

Honest tradeoffs of pushing the limit

Maximizing tokens per minute limit throughput requires walking a line:

• Higher concurrency raises utilization but increases tail latency and 429 risk.
• Prompt caching (via cache-control) reduces counted tokens but depends on provider support and cache hit rate.
• Automatic fallback to a secondary provider when the primary is degraded can sustain throughput. A gateway like n4n.ai performs this fallback transparently, so a rate-limited provider does not stall your pipeline. This is the only scenario where the effective TPM of your app exceeds the single-provider quota.
• Per-token metering (which n4n.ai exposes) lets you attribute cost precisely, but it does not change the physics.

There is no free lunch. The model’s FLOPS bound the maximum tokens per second; the quota bounds the average over a minute. Your job is to pack the window without tripping secondary limits.

Decisive takeaway

Stop quoting the provider’s TPM number in your capacity plans. Run a load test that mirrors your real prompt/completion ratio and concurrency, measure effective tokens per minute limit throughput, and design for 70% of that observed ceiling to absorb traffic spikes and retry backoff. Use prompt caching and batching where latency allows. If you operate across multiple providers, use a gateway that fails over automatically and meters per token so you can see the real number. The printed limit is a boundary; your throughput is what you engineer.