Concurrent request limits across inference providers

Concurrent request limits inference providers enforce are often the hidden bottleneck that breaks LLM apps in production. While most teams monitor requests per minute, the number of simultaneous in-flight calls frequently triggers 429s earlier than expected. This post compares how OpenAI, Anthropic, Google’s Gemini API, Azure OpenAI, and Groq define and throttle concurrency, then gives a concrete recommendation per use case.

How concurrency limits differ from RPM

Rate limits are usually expressed as requests per minute (RPM) or tokens per minute (TPM). Concurrency is the count of requests that have been sent but not yet received a response. A provider may allow 1,000 RPM but if each request takes 6 seconds, the sustained concurrency ceiling is roughly 100. Some providers publish a hard concurrent request cap; others leave it implicit via latency and RPM.

Understanding both matters because a bursty client (e.g., a web endpoint that fans out to 50 parallel summarization calls) will hit a concurrency wall even when its per-minute average is low.

OpenAI: tier-based RPM/TPM, implicit concurrency

OpenAI does not publish a separate “max concurrent requests” number. Instead, it uses tiered RPM and TPM quotas that scale with spend history. A new paid account (Tier 1) typically gets 500 RPM and 40,000 TPM for models like gpt-4o-mini, while Tier 5 can reach 10,000 RPM and 400,000 TPM on gpt-4o.

Capabilities: broad model roster, function calling, vision, fine-tuning. Price: per-token, billed on input/output separately. Latency: 300–1,500 ms for small prompts, longer for large generations. Ergonomics: official SDKs, mature tooling. Ecosystem: largest community and third-party integrations.

Because concurrency is implicit, you compute your safe parallel window as RPM / 60 * avg_latency_seconds. At 500 RPM and 2 s average latency, you can hold ~16 concurrent requests without breaching RPM.

Anthropic: explicit concurrent request tiers

Anthropic documents concurrent request limits directly. As of the latest published tiers, Claude API limits are:

• Tier 1: 5 concurrent requests, 50,000 TPM
• Tier 2: 20 concurrent requests, 100,000 TPM
• Tier 3: 50 concurrent requests, 300,000 TPM
• Tier 4: 100 concurrent requests, 1,000,000 TPM

Capabilities: Claude 3 family, 200K context, tool use. Price: per token, similar to OpenAI. Latency: competitive, often 400–1,200 ms. Ergonomics: SDKs with strict 429 on concurrency overflow. Ecosystem: growing, strong in long-context enterprise use.

The explicit cap means a single worker process should use a semaphore of size N matching its tier, or it will see rate_limit_error with retry_after hints.

import asyncio
from anthropic import Anthropic

client = Anthropic()
sem = asyncio.Semaphore(20)  # Tier 2 concurrent cap

async def call(prompt):
    async with sem:
        return await client.messages.create(
            model="claude-3-5-sonnet-20240620",
            max_tokens=512,
            messages=[{"role": "user", "content": prompt}]
        )

Google Gemini: RPM-centric

The Gemini API (AI Studio or Vertex) exposes RPM and TPM, not a concurrency integer. Free tier allows 60 RPM on gemini-1.5-pro, paid tier 360 RPM. Capabilities: native multimodal, 1M–2M token context. Price: per token, cheaper for large inputs. Latency: variable, 500–2,000 ms. Ergonomics: REST or SDK, less uniform across regions. Ecosystem: Google Cloud, BigQuery integration.

You derive concurrency as RPM / 60 * latency. At 360 RPM and 1.5 s latency, ~9 concurrent calls.

Azure OpenAI: provisioned throughput

Azure OpenAI does not publish a named concurrency limit; instead you deploy a model with a TPM quota (e.g., 240K TPM) and the service throttles when you exceed token throughput. Higher TPM deployments effectively permit more parallel calls. Capabilities: parity with OpenAI models inside Azure. Price: Azure billing, often enterprise agreements. Latency: comparable to OpenAI. Ergonomics: ARM templates, portal scaling. Ecosystem: enterprise compliance, VNet, private endpoints.

Groq: low latency, RPM-bound

Groq uses LPU hardware for very fast inference on open models (Mixtral, Llama 3). Free tier: 30 RPM, 14,400 TPM. Paid tiers raise RPM to 600+. Capabilities: limited model set, but sub-200 ms time-to-first-token. Price: per token, among the cheapest. Latency: 100–400 ms total. Ergonomics: OpenAI-compatible endpoint. Ecosystem: smaller, but popular for real-time UX.

Because latency is tiny, concurrency derived from RPM is high: 600 RPM / 60 * 0.3 s = 3 concurrent — but actual burst tolerance is better due to queueing.

Gateway abstraction

A gateway like n4n.ai can absorb these differences by exposing one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited or degraded. Instead of coding per-vendor semaphores, you set a client timeout and let the gateway spread load across providers with spare concurrency.

Head-to-head comparison

Client-side patterns for staying under limits

Whatever the provider, enforce a local bound:

import asyncio, openai

client = openai.AsyncOpenAI(max_retries=0)
conc = 16  # derived from RPM/latency
sem = asyncio.Semaphore(conc)

async def gen(p):
    async with sem:
        try:
            return await client.chat.completions.create(
                model="gpt-4o-mini", messages=[{"role":"user","content":p}]
            )
        except openai.RateLimitError as e:
            await asyncio.sleep(float(e.response.headers.get("retry-after", 1)))
            return await gen(p)

Use exponential backoff, jitter, and separate semaphores for read vs. write paths.

Which to choose

Prototype or general SaaS — OpenAI for breadth, or Anthropic if you need explicit concurrency control and long context. Both cover most needs.

High-volume, latency-sensitive UX — Groq for open models where sub-second response is mandatory; pair with a fallback to OpenAI on missing capabilities.

Multimodal or massive context — Gemini 1.5 Pro for 1M-token ingestion; watch the 360 RPM ceiling and batch offline jobs.

Regulated enterprise — Azure OpenAI for VNet, compliance, and TPM-based scaling; provision enough TPM to meet peak concurrency.

Multi-provider resilience — Use a gateway that honors client routing directives and forwards provider cache-control hints, so you stop writing per-vendor throttling and start declaring intent.

Concurrent request limits inference providers impose are not interchangeable. Map your latency budget to their published numbers before you ship, or your first traffic spike will teach you the hard way.