Understanding the real-world GPT-4o and Claude concurrent request limits separates a demo from a production LLM system. Published rate limits in requests per minute are misleading because they hide token-weighting and provider-side queuing; you need to measure saturation under your own traffic shape.
Why concurrency breaks before RPM does
Most engineers read the RPM number and assume they can open that many parallel connections. They can’t. A provider counts tokens per minute (TPM) separately, and a single GPT-4o call with a 2k-token prompt and 1k completion burns 3k of your TPM budget. Fire 50 concurrent requests and you’ll blow the TPM cap in under a second even if RPM looks fine.
Claude behaves similarly. Anthropic’s limits are token-aware, and their API returns 429s when either the per-minute request count or the concurrent in-flight count crosses an internal threshold. The practical GPT-4o and Claude concurrent request limits are therefore derived from token math, not a fixed socket cap.
If you size your worker pool by RPM alone, you’ll see latency cliffs and sporadic 429s that are hell to debug. The fix is to treat concurrency as a token-rate problem.
Token math: the real limiter
Providers expose RPM and TPM. Convert those to a concurrency ceiling using your average request weight.
Example: GPT-4o tier with 30,000 TPM. Your prompts average 1,500 tokens and you cap output at 500. Each request costs ~2,000 tokens. Theoretically you can sustain 15 requests per minute if perfectly serialized. But if you allow 20 concurrent connections, each finishing in 2 seconds, you inject 20×2,000 = 40,000 tokens in a 2-second window—equivalent to 1.2M TPM. The provider’s token bucket drains and you get 429s.
OpenAI GPT-4o
OpenAI documents tiered RPM and TPM limits that increase as your account ages and spends. A new paid tier might see 500 RPM and 30k TPM; higher tiers scale to thousands of RPM and millions of TPM. There is no stated “max concurrent connections” because the system sheds load via 429s when TPM or RPM is exceeded. In practice, each request holds model compute proportional to output tokens, so concurrency is bounded by (TPM / avg_tokens_per_request) / window_seconds.
Anthropic Claude
Anthropic specifies per-minute token limits and request counts, and historically signals a soft concurrent request ceiling via retry-after headers. Their guidance suggests keeping concurrency low (single digits to tens) for stable latency. The GPT-4o and Claude concurrent request limits thus share a theme: the visible numbers are per-minute, the invisible constraint is simultaneous inflight tokens.
Building a benchmark that reflects production
Synthetic “hello world” prompts produce fantasy numbers. Your real traffic has a distribution of prompt sizes, output caps, and temperature settings. Capture a sample of 1,000 real requests from logs, strip PII, and replay them.
Use an async client with a semaphore to control concurrency. Below is a minimal Python harness for GPT-4o:
import asyncio, time
from openai import AsyncOpenAI
client = AsyncOpenAI()
semaphore = asyncio.Semaphore(20) # tune this
async def call(prompt_tokens: int, max_out: int):
async with semaphore:
start = time.monotonic()
try:
await client.chat.completions.create(
model="gpt-4o",
messages=[{"role":"user","content":"x" * prompt_tokens}],
max_tokens=max_out,
)
return time.monotonic() - start, None
except Exception as e:
return None, str(e)
async def main(n, pt, mo):
tasks = [call(pt, mo) for _ in range(n)]
return await asyncio.gather(*tasks)
# run: asyncio.run(main(100, 1500, 500))For Claude, swap in the Anthropic SDK:
from anthropic import AsyncAnthropic
ac = AsyncAnthropic()
async def claude_call(pt, mo):
async with semaphore:
await ac.messages.create(
model="claude-3-5-sonnet-latest",
max_tokens=mo,
messages=[{"role":"user","content":"y"*pt}],
)Drive the semaphore value from an outer loop: 5, 10, 20, 50, 100. Record latency and errors per level. Replay real token sizes instead of constants when you can.
Streaming changes the connection profile
Non-streaming calls hold a connection until the full output is generated. Streaming keeps the connection open and trickles tokens, but the request still occupies a concurrent slot on the provider side until the final token. If your client uses streaming for UX, your connection count is identical to non-streaming for rate-limit purposes, but your own socket pool may exhaust sooner. Test with the same streaming flag you ship.
Metrics that actually matter
Raw throughput (requests/sec) is a vanity metric. You want:
- p50 / p95 latency per concurrency level
- Error rate (429, 5xx, timeouts)
- Effective goodput: successful requests that returned valid output per minute
A typical result set looks like:
{
"concurrency": 20,
"sent": 200,
"success": 198,
"p50_ms": 820,
"p95_ms": 2100,
"errors": {"429": 2}
}When p95 latency grows superlinearly while error rate stays low, you are hitting compute saturation, not rate limits. When 429s spike, you’ve exceeded the provider’s token or request budget. The true GPT-4o and Claude concurrent request limits for your workload are the highest concurrency where p95 stays under your SLO and 429 rate is <1%.
Tradeoffs of self-run load tests
Running this from a single cloud VM is cheap but biased. Provider edge caches, regional capacity, and your own NAT IP reputation affect results. If you benchmark from us-east-1 and your users are in APAC, you’ll miss cross-region latency.
Distributed load generators cost more and add coordination overhead. For most teams, a single-region test against production-like prompts is enough to size the client pool; global tuning comes later via real traffic observation.
Another tradeoff: cost. Firing 1,000 GPT-4o calls with 2k tokens each burns real dollars. Sample narrowly, then extrapolate.
Gateway patterns for surviving limits
Client-side backoff is necessary but not sufficient. When OpenAI or Anthropic degrades, you need fallback. An inference gateway such as n4n.ai can automatically reroute to a secondary provider when the primary returns 429 or timeout, honoring your routing directives and forwarding cache-control hints. That removes the need to hand-roll fallback logic, but you still must set concurrency caps based on the benchmark above—the gateway won’t invent capacity that isn’t there.
If you don’t use a gateway, implement a token-bucket limiter per provider and a circuit breaker. A minimal Python token bucket:
import time, asyncio
class TokenBucket:
def __init__(self, rate, capacity):
self.rate = rate
self.capacity = capacity
self.tokens = capacity
self.updated = time.monotonic()
self.lock = asyncio.Lock()
async def consume(self, n=1):
async with self.lock:
now = time.monotonic()
self.tokens += (now - self.updated) * self.rate
self.tokens = min(self.tokens, self.capacity)
self.updated = now
if self.tokens >= n:
self.tokens -= n
return True
return FalseContinuous benchmarking in CI
Provider limits drift. A monthly curl-loop against a canary prompt catches regression. Run a 5-minute ramp at 70% of your estimated ceiling and alert if p95 moves >20%. This is cheaper than a full replay and spots backend changes.
# simple hourly canary via xargs concurrency
for c in 10 20 30; do
seq 50 | xargs -P $c -I{} curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
https://api.openai.com/v1/chat/completions -H "Authorization: Bearer $KEY" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"ping"}]}'
doneNot a substitute for token-weighted tests, but a smoke signal.
Decisive takeaway
Stop trusting the RPM column. Measure your own token-weighted concurrency ceiling for GPT-4o and Claude using a replay of real prompts, ramp the semaphore, and plot p95 against error rate. Set your production worker pool to 70% of the concurrency where 429s begin, and put a fallback path in front of the model calls. That combination—benchmark, cap, fallback—is what keeps an LLM feature online when the provider’s silent concurrency limit bites.