An llm api rate limits token bucket is a throttling construct that meters both request count and token spend by dripping tokens into a finite bucket at a fixed rate, where each call withdraws tokens equal to its billed cost. Unlike a naive requests-per-minute ceiling, it permits short bursts up to bucket capacity while enforcing a long-term average rate.
How the token bucket algorithm works
The bucket holds capacity tokens. A background process adds refill_rate tokens per second (or per window). When a request arrives, the system checks if enough tokens exist for the estimated cost (prompt + max completion tokens). If yes, it deducts and serves; if no, it rejects with 429 or queues.
Cost is not just request count
For LLMs, a token bucket usually tracks tokens, not requests. A 10k-token prompt costs more than a 50-token ping. Some gateways run two coupled buckets: one for requests-per-minute (RPM) and one for tokens-per-minute (TPM). The stricter wins.
import time
class TokenBucket:
def __init__(self, capacity: float, refill_per_sec: float):
self.capacity = capacity
self.tokens = capacity
self.refill = refill_per_sec
self.last = time.monotonic()
def consume(self, cost: float) -> bool:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill)
self.last = now
if self.tokens >= cost:
self.tokens -= cost
return True
return FalseThis local simulator mirrors what a gateway does centrally across many clients, except the gateway must use atomic operations and per-API-key state.
Why LLM APIs use token buckets instead of flat RPM
A flat “60 requests/minute” cap is meaningless when one request processes 100k tokens and another processes 100. GPU memory and scheduler time correlate with tokens, not HTTP hits. The llm api rate limits token bucket aligns throttling with underlying compute.
Multi-tenant isolation
Providers partition buckets per organization, per model, and sometimes per endpoint. A burst on gpt-4o does not starve your text-embedding-3 traffic because the buckets are separate.
Provider degradation and cache hints
When an upstream provider is degraded, a gateway may shed load by tightening the effective bucket. If you call through an aggregation layer, the visible limit can shift mid-session. Gateways that forward provider cache-control hints can also skip token deduction for cache hits, so a repeated long prompt may cost zero against your bucket. Per-token usage metering in the response lets you reconcile the actual deduction with your estimate.
Concrete gateway response shape
A well-behaved LLM gateway returns headers so you can predict rejection before sending a large payload. Typical OpenAI-compatible responses include:
curl -i https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{"model":"meta-llama/llama-3-70b","max_tokens":512}'{
"x-ratelimit-limit-tokens": "100000",
"x-ratelimit-remaining-tokens": "42300",
"x-ratelimit-reset-tokens": "12.4",
"x-ratelimit-limit-requests": "600",
"x-ratelimit-remaining-requests": "588",
"x-ratelimit-reset-requests": "30"
}If your estimated token cost exceeds remaining, preemptively wait reset seconds or shed lower-priority traffic.
Client-side handling patterns
Naive immediate retry amplifies thundering herds. Use capped exponential backoff with full jitter.
async function callWithBackoff(req: () => Promise<Response>, maxRetries = 5) {
let attempt = 0;
while (true) {
const res = await req();
if (res.status !== 429) return res;
if (attempt >= maxRetries) throw new Error("rate limited exhausted");
const retryAfter = Number(res.headers.get("retry-after") ?? "1");
const jitter = Math.random() * retryAfter;
await new Promise(r => setTimeout(r, (retryAfter + jitter) * 1000));
attempt++;
}
}Honor client routing directives
If your gateway honors routing hints (e.g., x-n4n-route: provider-a), know that each provider behind it has its own llm api rate limits token bucket. A fallback to a secondary provider still consumes that provider’s bucket.
Common misconceptions
“Rate limit” equals requests per minute
False. For LLM endpoints the binding constraint is usually tokens-per-minute. A 200-RPM limit with a 20k-TPM limit means you can hit 200 tiny requests or far fewer large ones.
Burst capacity is infinite
The bucket capacity is finite. If capacity is 100k tokens and you fire ten 30k-token requests concurrently, only three succeed; the rest get 429. Burst merely absorbs brief spikes up to capacity.
A 429 means the service is down
It means your bucket is empty. The platform is healthy. Inspect x-ratelimit-remaining-* before assuming global outage.
Token buckets are only for inbound requests
Gateways also use egress buckets toward upstream providers. Your call may pass your own bucket but be rejected by the provider bucket downstream. An OpenAI-compatible endpoint that fronts 240+ models, such as n4n.ai, can mask some of this with automatic fallback when a provider is rate-limited or degraded, but your client still must handle the aggregate 429 from the edge.
Retry-After is always present
Some gateways return retry-after; others return only x-ratelimit-reset-*. Parse both, default to backoff.
Cached prompts are free everywhere
Only if the gateway forwards cache-control and the provider supports prompt caching. Otherwise the full prompt cost is deducted.
Designing for the bucket
Engineers should size workloads to the sustained refill rate, not peak capacity. If your bucket refills at 50k TPM and you average 60k TPM, you will periodically stall regardless of burst headroom.
Prefill token estimates using tiktoken or model-specific counts before sending. If estimate exceeds remaining, either wait or reduce max_tokens.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
prompt_cost = len(enc.encode("Long system prompt..."))
max_completion = 1024
if bucket.consume(prompt_cost + max_completion):
# proceed
else:
# deferOperational checklist
- Treat token and request buckets as independent dimensions.
- Read rate-limit headers and branch locally before sending large payloads.
- Backoff with jitter; never spin on 429.
- Size for refill rate, not capacity.
- Remember that the llm api rate limits token bucket you see at the edge may be an aggregate of multiple upstream buckets.
- Use per-token metering to audit actual deductions versus estimates.