Load testing checklist before an LLM product launch

Most teams treat load testing as a late-stage checkbox until their LLM product launch load testing checklist becomes a 2am scramble. LLM endpoints behave nothing like traditional REST APIs: token streaming, variable latency, and provider rate limits turn capacity planning into a different discipline. Below is the checklist we run before shipping anything that fronts a model.

1. Model your real traffic shape, not just QPS

Requests per second is a vanity metric for LLM services. What matters is concurrent in-flight requests, input token distribution, and max output tokens, because providers bill and throttle on tokens, not HTTP calls. Pull your actual prompt lengths from logs or proxy records and build a weighted distribution.

import random
# Observed prompt token lengths from production logs
prompt_tokens = random.choices([32, 128, 512, 1024], weights=[0.5, 0.3, 0.15, 0.05])
max_output = random.choice([256, 512, 1024])

Run your load generator against that distribution, not a fixed 1KB payload. A system that handles 100 QPS of 32-token prompts will fall over at 20 QPS of 1K-token prompts due to KV-cache pressure and TPM limits. Your LLM product launch load testing checklist must start with this reality.

2. Test streaming and timeout handling under load

Nearly every chat or agent response streams tokens. If your client buffers the entire response or has a naive read timeout, a slow provider will hang connections and exhaust your worker pool. Load test with stream: true and assert that you consume bytes incrementally.

from locust import HttpUser, task, between

class LLMUser(HttpUser):
    wait_time = between(1, 5)
    @task
    def chat(self):
        with self.client.post("/v1/chat/completions",
            json={"model":"gpt-4o","messages":[{"role":"user","content":"load test"}],"stream":True},
            stream=True) as r:
            for line in r.iter_lines():
                if line:
                    pass  # consume stream

Set explicit client-side timeouts for time-to-first-token (TTFT) and inter-token gap. Under load, a provider may queue requests; your timeout should trip and retry, not block. This is a non-negotiable line item on any LLM product launch load testing checklist.

3. Validate provider rate limits and fallback paths

Every provider publishes RPM and TPM caps; you will hit them. Send a sustained burst that exceeds your quota and confirm your stack degrades gracefully. Inspect the rate-limit response headers and ensure your code reads them.

curl -i https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"claude-3","messages":[]}'
# Observe: x-ratelimit-remaining-requests: 0

If you sit behind an OpenRouter-class gateway such as n4n.ai, automatic fallback masks provider 429s when a vendor is degraded, but the gateway itself can still throttle at the account level. Test the full chain: provider limit → gateway fallback → your retry. A missing Retry-After parse will cause tight loops that get you banned.

4. Measure tail latency at target concurrency, not average

Average latency lies. At 50 concurrent streams, p50 TTFT might be 400ms while p99 is 12s because the model scheduler packed batches. Capture percentile histograms for TTFT and tokens-per-second (TPS) at your projected peak concurrency.

Use a metrics sink (Prometheus, OTel) and graph p95/p99 over the test duration. If p99 TTFT climbs above your product’s acceptable wait, you need request prioritization or a smaller model for long-tail traffic. This belongs on the LLM product launch load testing checklist because users feel the tail, not the mean.

5. Exercise your retry and backoff logic with chaos

Most LLM client libraries ship naive retries. Under load, a 500 or 429 should trigger jittered exponential backoff, not a tight for loop. Write a fault-injecting proxy that returns 429 20% of the time and verify your client survives.

async function retryWithBackoff(fn: () => Promise<any>, max=5) {
  for (let i=0; i<max; i++) {
    try { return await fn(); }
    catch (e) {
      if (i===max-1) throw e;
      const delay = Math.min(1000*2**i, 30000) + Math.random()*200;
      await new Promise(r=>setTimeout(r,delay));
    }
  }
}

Without jitter, synchronized retries create thundering herds that amplify the outage. Run this chaos test weekly, not just before launch. It is the cheapest insurance on the list.

6. Load test the surrounding orchestration

An LLM call is rarely the only bottleneck. RAG pipelines hit vector databases; agents call tools; guardrails run classifiers. Load test the full request graph with realistic branching, because a 5ms vector query that serializes 200 concurrent requests becomes the limiter.

Spin up your orchestrator with the same synthetic traffic from step 1 and profile each span. If your tool schema validation adds 300ms of blocking CPU, that shows up as inflated TTFT under load. The LLM product launch load testing checklist must include the mesh, not just the model.

7. Verify cost metering and quota guards

Token spend is continuous and unbounded under load. Confirm your metering matches provider invoices within 1% and that daily budgets trigger alerts before ruin. A misconfigured limit can drain a month of runway in an hour.

{
  "alerts": {
    "token_budget_daily": 50000000,
    "notify": "pagerduty"
  }
}

Wire these guards into your load test: force a 10x traffic spike and assert the circuit breaker trips and returns a clean 429 to clients instead of silently accruing debt. This is where many launches die quietly.

8. Run a soak test before flip

A 30-minute load test hides memory leaks, connection pool exhaustion, and provider long-term throttling. Run a 24-hour soak at 70–80% of projected peak with streaming enabled. Watch for creeping p99 or rising error rate after hour 6.

This final step on the LLM product launch load testing checklist catches what spike tests miss. If the system is stable at hour 23, you are ready to redirect real users.

Summary

Run this LLM product launch load testing checklist verbatim and you will avoid the outages that fill post-mortem blogs.