Providers advertise requests per minute cap effective throughput as if it were a bandwidth spec, but the number of requests you can fire in sixty seconds tells you little about how many useful completions you’ll get. The gap between the rated limit and what your system actually sustains comes from latency, retries, and payload shape. If you size infrastructure off the raw RPM number, you will either over-provision clients or starve your own pipeline.
What an RPM cap actually restricts
A requests-per-minute limit is a counter on the provider’s control plane. It increments on each HTTP request that reaches the edge, regardless of whether the request succeeds, returns garbage, or takes ten seconds to stream.
Attempts, not completions
The cap does not care about tokens generated. A 60 RPM limit permits 60 tiny health-check calls or 60 massive document summarizations with equal count. Effective throughput in any production system is measured in finished, valid responses per minute, not connection attempts. When you measure requests per minute cap effective throughput, you must count finished, valid responses.
Latency is the hidden tax
If each request takes 2 seconds of round-trip time, a single thread can only issue 30 requests per minute, never hitting a 60 RPM cap. You need concurrency to approach the limit. But concurrency introduces its own overhead: connection pools, context switching, and timeout handling.
Little’s Law makes this concrete: N = λ * W, where N is concurrent in-flight requests, λ is completion rate (requests/sec), and W is mean latency per request. To sustain λ=1 req/sec (60 RPM) at W=2s, you need N=2. At W=5s, you need N=5. Ignore this and your client threads block before the cap is reached.
Measuring requests per minute cap effective throughput
To get a real number, build a load test that mirrors your traffic. A synthetic loop that sleeps 1/RPM seconds between calls measures nothing useful because it assumes zero latency and infinite patience.
Define success precisely
Count a request as effective only if it returns HTTP 200, a parseable completion, and meets your latency SLO. Dropped connections, 429s, and truncated streams are zeros. A 200 with an empty choices array is also a zero—your application got no work.
Load shape: Poisson, not metronome
Production traffic arrives in bursts. Use a Poisson arrival process with a target mean rate near the cap. This exposes queueing delays and retry storms that a fixed-interval sender hides. A metronome sender at exactly 60 RPM will never trigger the burst behavior that drops you into provider throttling.
A minimal concurrent benchmark
Below is a Python asyncio client that hammers an OpenAI-compatible endpoint with controlled concurrency and reports completed requests per minute.
import asyncio, time, aiohttp, random
BASE = "https://api.example.com/v1/chat/completions"
HEADERS = {"Authorization": "Bearer KEY", "Content-Type": "application/json"}
PAYLOAD = {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Say hi in 5 words."}],
"max_tokens": 20,
}
async def worker(session, sem, results):
async with sem:
start = time.monotonic()
try:
async with session.post(BASE, json=PAYLOAD, headers=HEADERS, timeout=30) as resp:
if resp.status == 200:
data = await resp.json()
if data.get("choices"):
results.append(time.monotonic() - start)
except Exception:
pass
async def run(rpm_target, duration_sec=60):
rate = rpm_target / 60.0
sem = asyncio.Semaphore(50) # concurrency bound
results = []
async with aiohttp.ClientSession() as session:
tasks = []
end = time.monotonic() + duration_sec
while time.monotonic() < end:
await asyncio.sleep(random.expovariate(rate))
tasks.append(asyncio.create_task(worker(session, sem, results)))
await asyncio.gather(*tasks)
effective = len(results) / duration_sec * 60
print(f"Completed {len(results)} reqs, avg latency {sum(results)/len(results):.2f}s")
print(f"Effective throughput: {effective:.1f} requests/min")
asyncio.run(run(60))This script sends at a mean rate of 60 RPM but caps concurrent in-flight at 50. If the endpoint honors the cap, you’ll see completed count near 60 minus failures. If latency spikes, completions drop because fewer workers finish inside the window.
Variables that quietly crush throughput
Token limits and response size
A 60 RPM cap with max_tokens=2000 yields far fewer useful completions per minute than the same cap with max_tokens=50, because each request occupies the model longer. Throughput in tokens per minute is often the metric that matters. A summarization task generating 1500 tokens at 4s latency needs concurrency of ~4 to saturate a 60 RPM cap; a classification task generating 10 tokens at 300ms needs concurrency of ~0.3, meaning one worker is enough.
Retries and backoff
Naive clients retry 429s immediately, creating a thundering herd that pushes the provider into deeper degradation. Implement exponential backoff with jitter. Each retry consumes a slot in your RPM budget without producing a completion. Worse, if you count attempts rather than successes, your logs will show “we hit 60 RPM” while effective throughput is near zero.
Streaming changes the timeline
With streaming, the HTTP request finishes when the first token arrives, but your application logic may wait for the last token. If you count “request completed” at stream open, your effective throughput looks higher than actual usable output. Count stream closure. For long completions, the gap between first and last token can be seconds, and that time still holds a connection and a worker.
Gateway fallback and the RPM illusion
When a primary provider hits its RPM wall, your client gets 429s. Those rejected attempts still counted against your sent rate but produced zero value.
A gateway such as n4n.ai that provides automatic fallback when a provider is rate-limited or degraded alters the calculus: your client sees fewer hard rejects because the gateway routes to a secondary model. The requests per minute cap effective throughput on your side stays high because the gateway absorbs the control-plane rejection and returns a completion from elsewhere. This is not free—cross-model fallback changes output formatting—but it preserves pipeline momentum.
Honoring cache-control
Some gateways forward provider cache-control hints. If your prompt prefix is cached, the secondary provider may still benefit, reducing latency and improving effective RPM headroom. That means your measured effective throughput under fallback can exceed what the primary provider alone would deliver at the same cap.
Tradeoffs in benchmark design
Conservative vs aggressive concurrency
Set concurrency too low and you never reach the cap; too high and you trigger provider-side connection limits or self-inflicted timeouts. Start at 2× expected RPM and tune via latency percentiles. Watch p95 latency: if it climbs while completed RPM stalls, you are queueing, not working.
Cost of measurement
Running a 60-minute soak at 1000 RPM against a paid model costs real money. Use small max_tokens and cheap models for shape testing, then validate with one production-sized burst. If your inference layer reports per-token usage metering, you can normalize effective throughput by cost instead of raw counts—n4n.ai exposes per-token usage metering, which lets you convert effective RPM into cost per completed task without building your own accounting.
Client vs server clock
Your load generator’s wall clock differs from the provider’s sliding-window counter. A provider may use a 60-second rolling window, not a fixed cron minute. Your benchmark should measure over a window longer than the provider’s to avoid edge effects where the counter resets mid-burst.
Decisive takeaway
Treat the published requests per minute cap effective throughput as an upper bound on request attempts, not a promise of work done. Measure your own effective throughput as successful, valid completions per minute under Poisson load with realistic concurrency and payloads. Build clients that adaptively saturate the cap, back off on 429s, and consider gateway-level fallback to keep the pipeline moving when a single provider degrades. The teams that instrument this correctly stop guessing and start shipping predictable LLM features.