Tokens-per-minute limits LLM providers enforce cap the total input and output tokens a client can consume within a 60-second sliding or fixed window. Unlike requests-per-minute, which counts calls, TPM tracks the actual computational payload—every prompt token and generated token eats the same budget. If you ship LLM features, understanding tokens-per-minute limits LLM providers is non-negotiable because a single long RAG completion can exhaust your quota faster than hundreds of tiny calls.
What Tokens-Per-Minute Limits Actually Mean
A token is the atomic unit of LLM billing and compute—roughly four characters of English text, or a subword for code. Providers expose rate limits along two axes: requests-per-minute (RPM) and tokens-per-minute (TPM). TPM is the sum of billable tokens processed across all requests in the window. Some providers split input TPM and output TPM; others use a combined pool.
For example, OpenAI treats TPM as a shared pool for a model tier, while Anthropic publishes separate input and output token rates. The limit is enforced per organization, not per API key, though enterprise contracts may carve out dedicated capacity.
How Providers Enforce TPM
Rolling vs Fixed Windows
Most gateways use a rolling window: at any moment, the sum of tokens in the last 60 seconds must stay under the cap. A few use fixed calendar-minute buckets, which cause thundering-herd spikes at minute boundaries. Know which you face by inspecting reset headers.
Signal Headers
OpenAI-compatible APIs return rate-limit state in response headers. You should read these instead of guessing.
curl -i https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'Response headers include:
{
"x-ratelimit-limit-tokens": "150000",
"x-ratelimit-remaining-tokens": "149872",
"x-ratelimit-reset-tokens": "12"
}The reset value is seconds until the window clears. If you ignore it, you will 429 on the next large request.
Why TPM Beats RPM For Capacity Planning
RPM hides the real constraint. A chatbot sending 200-byte prompts with 50-token answers scales differently than a document pipeline emitting 4k-token summaries. The latter hits TPM long before RPM.
Consider a batch job: 1,000 documents, each 3,000 input tokens, producing 300 output tokens. Total tokens = 3.3M. At a 150k TPM limit, that batch needs at least 22 minutes of pure throughput, ignoring overhead. RPM might allow 3,000 requests per minute, but TPM strangles you at ~50 concurrent docs.
Tokens-per-minute limits LLM providers publish are often soft—they throttle, not hard-block—but sustained overuse gets your org flagged.
A Concrete Summarization Workload
Estimating Token Demand
Use a tokenizer locally to avoid blind spots. tiktoken for OpenAI models, or provider-specific equivalents.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
docs = ["..." * 1000 for _ in range(1000)] # fake corpus
total_in = sum(len(enc.encode(d)) for d in docs)
avg_out = 300
total_tokens = total_in + avg_out * len(docs)
print(f"Need {total_tokens} tokens across windows")If total_in is ~3M and output ~300k, you need ~3.3M tokens. At 150k TPM, schedule across 22 windows.
Throttling With A Token Bucket
Implement a simple async gate that releases based on token cost, not call count.
import asyncio, time
class TPMGate:
def __init__(self, tpm_limit: int, window: int = 60):
self.limit = tpm_limit
self.window = window
self.used = 0
self.start = time.monotonic()
async def consume(self, tokens: int):
while True:
now = time.monotonic()
if now - self.start >= self.window:
self.used = 0
self.start = now
if self.used + tokens <= self.limit:
self.used += tokens
return
await asyncio.sleep(0.5)
gate = TPMGate(150_000)
async def call_llm(prompt_tokens: int, gen_tokens: int):
await gate.consume(prompt_tokens + gen_tokens)
# actually call API hereThis pattern keeps you under the cap without polling headers constantly.
Common Misconceptions About Tokens-Per-Minute Limits LLM Providers
“TPM and RPM are interchangeable.” False. A limit of 10k RPM with 10k TPM means you can make ten thousand one-token calls, or one ten-thousand-token call. They constrain orthogonal dimensions.
“Streaming responses dodge TPM.” The token meter runs on usage, not latency. Streamed tokens count as they finalize.
“Prompt caching makes cached tokens free.” Some providers discount cached input tokens (e.g., 0.1x), but they still occupy the TPM budget unless the header explicitly exempts them. Always check x-ratelimit-* after a cached call.
“Limits are per key, so I can shard.” Most providers meter per organization or per project. Spawning keys does not raise the ceiling.
“Bursting above TPM is tolerated.” At best you get a 429 with retry-after; at worst your throughput is shaped for minutes.
Gateway Normalization Of TPM
When you route through an inference gateway, the underlying tokens-per-minute limits LLM providers impose become a fragmented mess: each backend uses different header names, window lengths, and separate input/output pools. A gateway such as n4n.ai normalizes these into a single OpenAI-compatible ratelimit contract, auto-falls back to a secondary provider when TPM is exhausted, and meters per-token usage so you get one bill and one backpressure signal. That abstraction is the only sane way to scale across 240+ models without writing per-vendor throttlers.
Patterns To Survive Real Limits
- Pre-compute token counts with a local tokenizer before sending. Never guess.
- Use batch endpoints where available; they often have separate, higher TPM tiers.
- Honor cache-control hints to reduce repeat input token cost.
- Exponential backoff with jitter on 429, reading
retry-afterand ratelimit-reset. - Shard by workload type, not by key: separate low-latency chat from bulk extract so one noisy neighbor doesn’t starve the other.
If you treat tokens-per-minute limits LLM providers set as a first-class constraint in your client, you avoid midnight pages and silent truncation. Build the bucket, read the headers, and design for the window.