What are API rate limits and why providers enforce them

API rate limits are constraints that restrict how many requests a client can make to an API within a given time window, typically expressed as requests per minute (RPM), tokens per minute (TPM), or concurrent requests. Providers enforce these limits to protect service stability, ensure fair access across customers, and manage underlying compute costs. Understanding what are API rate limits and how they behave is essential for building reliable LLM-powered applications.

How rate limits work

Rate limits operate on sliding or fixed time windows. A fixed window resets at predictable intervals — say, at the top of every minute — while a sliding window tracks the rolling count over the last N seconds. Most providers expose limits through response headers so clients can adapt dynamically.

Common header patterns include:

# OpenAI-style headers
x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 497
x-ratelimit-reset-requests: 1.2s

# Anthropic-style headers
anthropic-ratelimit-requests-limit: 50
anthropic-ratelimit-requests-remaining: 48
anthropic-ratelimit-requests-reset: 2024-01-15T10:30:00Z

# Generic token-based headers
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-tokens: 198500
x-ratelimit-reset-tokens: 45s

When you exceed a limit, the server returns 429 Too Many Requests with a Retry-After header indicating seconds to wait. Well-behaved clients parse these headers and back off automatically rather than hammering the endpoint.

Token-level vs request-level limits

LLM APIs often enforce both request and token limits simultaneously. A single request consuming 50,000 tokens might hit your TPM ceiling while leaving RPM headroom unused. This distinction matters when batching: ten 2,000-token requests consume the same token budget as one 20,000-token request but cost more in request quota.

# Example: checking both limits before sending
async def can_send_request(headers: dict, estimated_tokens: int) -> tuple[bool, float]:
    rpm_remaining = int(headers.get("x-ratelimit-remaining-requests", "0"))
    tpm_remaining = int(headers.get("x-ratelimit-remaining-tokens", "0"))
    rpm_reset = float(headers.get("x-ratelimit-reset-requests", "60").rstrip("s"))
    tpm_reset = float(headers.get("x-ratelimit-reset-tokens", "60").rstrip("s"))

    if rpm_remaining <= 0:
        return False, rpm_reset
    if tpm_remaining < estimated_tokens:
        return False, tpm_reset
    return True, 0.0

Concurrency limits

Separate from rate limits, concurrency limits cap simultaneous in-flight requests. You might have 500 RPM but only 10 concurrent connections. This prevents a single client from monopolizing connection pools or overwhelming downstream model servers. Exceeding concurrency typically returns 429 immediately, even if your minute-level quota has headroom.

Why providers enforce rate limits

Protecting shared infrastructure

Model inference runs on finite GPU clusters. Without limits, a single runaway script — or a malicious actor — could saturate capacity and degrade latency for everyone. Rate limits act as a circuit breaker at the API gateway layer, shedding load before it reaches model servers.

Fair allocation across tiers

Providers segment customers into tiers (free, pay-as-you-go, enterprise) with different quotas. Rate limits enforce these contractual boundaries. A free-tier user getting 3 RPM prevents them from accidentally or intentionally consuming capacity reserved for paying customers.

Cost predictability

Token-based limits let providers map API usage directly to GPU-hour costs. If a model costs $0.002 per 1K tokens and you allow 200K TPM, that’s a predictable $0.40/minute maximum spend per customer. Unbounded usage breaks capacity planning and billing models.

Abuse mitigation

Rate limits raise the cost of scraping, model extraction attacks, and credential stuffing. They don’t stop determined attackers but they force distributed, slower attacks that are easier to detect and block at the WAF layer.

Concrete example: building a resilient client

Here’s a production-grade pattern for handling what are API rate limits in an async Python client. The key insight: respect Retry-After, implement exponential backoff with jitter, and track local state to avoid unnecessary 429s.

import asyncio
import random
import time
from dataclasses import dataclass
from typing import Optional
import httpx

@dataclass
class RateLimitState:
    rpm_remaining: int = 500
    tpm_remaining: int = 200000
    rpm_reset_at: float = 0
    tpm_reset_at: float = 0
    last_429_at: float = 0

class RateLimitedClient:
    def __init__(
        self,
        base_url: str,
        api_key: str,
        max_concurrent: int = 10,
        max_retries: int = 5,
        base_backoff: float = 1.0,
    ):
        self.client = httpx.AsyncClient(
            base_url=base_url,
            headers={"Authorization": f"Bearer {api_key}"},
            timeout=httpx.Timeout(60.0, connect=5.0),
            limits=httpx.Limits(max_connections=max_concurrent, max_keepalive_connections=max_concurrent),
        )
        self.state = RateLimitState()
        self.max_retries = max_retries
        self.base_backoff = base_backoff
        self._semaphore = asyncio.Semaphore(max_concurrent)

    def _update_from_headers(self, headers: httpx.Headers) -> None:
        now = time.monotonic()
        self.state.rpm_remaining = int(headers.get("x-ratelimit-remaining-requests", self.state.rpm_remaining))
        self.state.tpm_remaining = int(headers.get("x-ratelimit-remaining-tokens", self.state.tpm_remaining))

        rpm_reset = headers.get("x-ratelimit-reset-requests")
        if rpm_reset:
            self.state.rpm_reset_at = now + float(rpm_reset.rstrip("s"))

        tpm_reset = headers.get("x-ratelimit-reset-tokens")
        if tpm_reset:
            self.state.tpm_reset_at = now + float(tpm_reset.rstrip("s"))

    def _estimate_tokens(self, payload: dict) -> int:
        # Rough heuristic: 4 chars ≈ 1 token for English
        text = str(payload.get("messages", "")) + str(payload.get("prompt", ""))
        return max(len(text) // 4, 1)

    async def _wait_for_capacity(self, estimated_tokens: int) -> None:
        now = time.monotonic()
        wait_times = []

        if self.state.rpm_remaining <= 0 and self.state.rpm_reset_at > now:
            wait_times.append(self.state.rpm_reset_at - now)
        if self.state.tpm_remaining < estimated_tokens and self.state.tpm_reset_at > now:
            wait_times.append(self.state.tpm_reset_at - now)

        if wait_times:
            await asyncio.sleep(max(wait_times) + 0.1)  # small buffer

    async def post(self, endpoint: str, payload: dict) -> httpx.Response:
        estimated_tokens = self._estimate_tokens(payload)
        last_exception = None

        for attempt in range(self.max_retries + 1):
            await self._wait_for_capacity(estimated_tokens)

            async with self._semaphore:
                try:
                    response = await self.client.post(endpoint, json=payload)
                    self._update_from_headers(response.headers)

                    if response.status_code == 429:
                        self.state.last_429_at = time.monotonic()
                        retry_after = response.headers.get("Retry-After")
                        wait = float(retry_after) if retry_after else self._exponential_backoff(attempt)
                        await asyncio.sleep(wait)
                        continue

                    response.raise_for_status()
                    return response

                except httpx.RequestError as e:
                    last_exception = e
                    wait = self._exponential_backoff(attempt)
                    await asyncio.sleep(wait)

        raise last_exception or RuntimeError("Max retries exceeded")

    def _exponential_backoff(self, attempt: int) -> float:
        # Full jitter: random(0, 2^attempt * base)
        max_wait = self.base_backoff * (2 ** attempt)
        return random.uniform(0, min(max_wait, 60.0))

This client:

• Tracks local quota state from response headers to avoid sending doomed requests
• Respects both RPM and TPM limits with separate reset timers
• Uses a semaphore to enforce client-side concurrency limits
• Implements full-jitter exponential backoff on 429s and network errors
• Estimates token usage locally for proactive throttling

Common misconceptions

“Rate limits are just suggestions”

They’re hard constraints. Providers enforce them at the edge — often in API gateways or load balancers before requests reach application logic. You cannot negotiate around a 429 by adding headers or retrying immediately. The only valid response is waiting.

“I can just use multiple API keys to bypass limits”

Key rotation works until it doesn’t. Providers correlate usage by IP, fingerprint, billing account, and behavioral patterns. Coordinated key rotation across a pool is detectable and typically violates terms of service. At scale, it gets your entire account banned. Build proper backoff instead.

“Rate limits only apply to requests, not streaming”

Streaming responses consume token quota in real time. A 10,000-token streamed completion counts against your TPM exactly like a non-streamed response. Some providers also count each chunk as a request for RPM purposes. Check the documentation.

“Enterprise tiers have unlimited rate limits”

They have higher limits, not infinite ones. Enterprise contracts specify quotas (e.g., 10K RPM, 5M TPM) with overage terms. Unlimited would break the provider’s capacity planning. If you genuinely need more, you negotiate a custom capacity reservation — often with dedicated instances.

“Caching responses avoids rate limits”

Caching helps with your rate limit consumption, but the provider still sees each unique request. If you’re building a gateway or proxy, you need your own rate limiting layer in front of the upstream API. This is where a gateway like n4n.ai adds value: it enforces per-client quotas locally while managing upstream provider limits centrally, including automatic fallback when a provider is rate-limited or degraded.

“Rate limit headers are always accurate”

Headers reflect state at response time. Under high concurrency, multiple in-flight requests can all see remaining: 1 and all proceed, causing a burst of 429s. Treat headers as hints, not guarantees. The client-side semaphore and local tracking in the example above mitigate this.

Headers you should actually read

Not all providers expose the same headers. At minimum, handle these:

Parse defensively. Headers may be missing, malformed, or use different units across providers. Normalize to a common internal representation.

Testing rate limit behavior

Don’t discover limits in production. Test deliberately:

# Hammer a test endpoint to observe 429 behavior
for i in {1..20}; do
  curl -s -w "HTTP %{http_code} | Remaining: %{header_x-ratelimit-remaining-requests}\n" \
    -H "Authorization: Bearer $TEST_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}' \
    https://api.openai.com/v1/chat/completions
done

Automate this in CI for your client library. Verify:

• Correct Retry-After parsing (seconds vs HTTP-date)
• Backoff behavior under sustained 429s
• Header updates on successful responses
• Concurrency limiting works independently of rate limits

Summary

What are API rate limits? They’re the guardrails that keep shared LLM infrastructure stable and fairly allocated. You’ll encounter request limits (RPM), token limits (TPM), and concurrency limits — often simultaneously. Respect the headers, implement exponential backoff with jitter, track local quota state, and test against real 429 responses before shipping. The alternative is fragile code that fails catastrophically under load.