Load testing an LLM API with k6 requires a different mindset than benchmarking a CRUD service. Token generation is compute-bound, responses stream incrementally, and provider rate limits turn naive concurrency into a wall of 429s. This guide gives you a runnable k6 harness that captures time-to-first-token, completion throughput, and error rates under realistic load.
Step 1: Install k6 and set up credentials
Grab k6 from your package manager or run it in Docker. You do not need the k6 Cloud account to run local load tests; the open-source binary handles everything here.
# macOS
brew install k6
# or Docker, no install needed
docker pull grafana/k6Export your API key and target base URL as environment variables. Keeping them out of the script avoids accidental commits and lets you repoint the test without editing code.
export API_KEY="sk-..."
export BASE_URL="https://api.openai.com/v1"If you run in Docker, pass them with -e flags. The script below reads __ENV.API_KEY and __ENV.BASE_URL.
Step 2: Write a minimal non-streaming smoke test
Start with a single virtual user (VU) hitting the chat completions endpoint. This validates auth, model name, and response shape before you layer on load.
import http from 'k6/http';
import { check, sleep } from 'k6';
const BASE = __ENV.BASE_URL || 'https://api.openai.com/v1';
const KEY = __ENV.API_KEY;
export default function () {
const res = http.post(`${BASE}/chat/completions`, JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Say hello in five words.' }],
max_tokens: 20,
}), {
headers: {
'Authorization': `Bearer ${KEY}`,
'Content-Type': 'application/json',
},
});
check(res, { 'status 200': (r) => r.status === 200 });
sleep(1);
}Run it with k6 run script.js. A green status check and zero script errors means the contract is correct. Do not skip this—debugging a 401 under 50 VUs wastes time.
Step 3: Capture time-to-first-token with streaming
The defining latency metric for an LLM endpoint is not total request time; it is how long the user waits before the first token appears. Set stream: true and read res.timings.firstByte. k6 does not expose incremental SSE frames natively, but first-byte timing is a solid proxy for time-to-first-token when the server flushes immediately.
import http from 'k6/http';
import { Trend } from 'k6/metrics';
export const ttft = new Trend('time_to_first_token_ms');
export default function () {
const res = http.post(`${BASE}/chat/completions`, JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Write a haiku about latency.' }],
stream: true,
max_tokens: 30,
}), {
headers: {
'Authorization': `Bearer ${KEY}`,
'Content-Type': 'application/json',
},
responseType: 'text',
});
ttft.add(res.timings.firstByte);
}For non-streaming requests you can parse the usage field to count tokens. With streaming, the usage is often in the final SSE event; if you need exact counts, disable stream in a separate metric pass or estimate from character length (≈4 chars/token for English).
Step 4: Add token throughput and error counters
A single latency number hides whether the model is actually producing output under load. Add a Counter for completion tokens and a rate-limited error tracker.
import { Counter, Rate } from 'k6/metrics';
export const completionTokens = new Counter('completion_tokens');
export const rateLimited = new Rate('rate_limited_429');
export default function () {
const res = http.post(`${BASE}/chat/completions`, JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Explain TCP in one sentence.' }],
max_tokens: 50,
}), {
headers: { 'Authorization': `Bearer ${KEY}`, 'Content-Type': 'application/json' },
});
if (res.status === 429) {
rateLimited.add(1);
return;
}
rateLimited.add(0);
try {
const body = res.json();
completionTokens.add(body.usage?.completion_tokens ?? 0);
} catch (e) { /* ignore parse errors on bad resp */ }
}Track completion_tokens per second at the dashboard level by dividing total counter by test duration. That gives you real generation throughput, not just request rate.
Step 5: Configure a realistic load profile
Static 10 VUs is not a load test; it is a warm-up. Use a ramping VU scenario to find the knee where latency balloons or errors spike.
export const options = {
scenarios: {
ramp: {
executor: 'ramping-vus',
startVUs: 1,
stages: [
{ duration: '30s', target: 5 },
{ duration: '1m', target: 20 },
{ duration: '30s', target: 20 },
{ duration: '30s', target: 0 },
],
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
time_to_first_token_ms: ['p95<4000'],
rate_limited_429: ['rate<0.05'],
},
};The thresholds encode your SLO: under 1% total HTTP failures, p95 time-to-first-token under 4 seconds, and fewer than 5% of calls hitting 429. Adjust based on the model class—a 70B weight needs looser bounds than a 7B.
Step 6: Run the test and verify success
Execute locally or in Docker. If using Docker, mount the script and pass env vars.
k6 run -e API_KEY=$API_KEY -e BASE_URL=$BASE_URL llm-load.jsSuccess criteria are explicit:
- Process exit code
0(k6 exits non-zero if any threshold fails). http_req_failedrate printed in summary is below 0.01.time_to_first_token_msp95 meets the threshold.- No
429storm:rate_limited_429rate under 0.05.
If thresholds fail, do not immediately raise VUs. Inspect the bottleneck: provider quota, network egress, or your own client-side connection reuse. k6 reuses TCP connections by default; if you see socket errors, check http_req_connecting timings.
Step 7: Swap in a multi-provider gateway
When you move from one provider to a routing layer, the test logic should not change. If you point the same script at a gateway such as n4n.ai, which exposes a single OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, you only change BASE_URL and API_KEY. The chat completions shape is identical, so the streaming and token metrics still hold. This lets you load test routing directives and cache-control hints without maintaining forked scripts.
export BASE_URL="https://api.n4n.ai/v1"
export API_KEY="your-gateway-key"
k6 run -e API_KEY=$API_KEY -e BASE_URL=$BASE_URL llm-load.jsHonor the gateway’s cache-control by passing extra_headers in the post options if you want to measure cache hit rates under repeated prompts.
Step 8: Analyze and iterate
Raw k6 output is enough for a pass/fail, but real signal lives in the trends. Plot time_to_first_token_ms and completion_tokens per VU stage. Three patterns matter:
- Linear latency rise with VUs — you are hitting a hard compute ceiling. Reduce concurrency or use a smaller model.
- Sudden 429 cliff — provider rate limit. Implement exponential backoff in the script (sleep on 429) to simulate real client behavior, then re-test.
- Flat throughput, rising errors — gateway or load balancer saturation, not the model. Check upstream health.
A practical tip: tag requests with model name so you can compare gpt-4o-mini vs mixtral-8x7b in the same run.
import { tag } from 'k6/http';
http.post(url, body, { tags: { model: 'gpt-4o-mini' } });k6 aggregates by tag automatically.
Step 9: Clean up and productionize
Parameterize prompts from a CSV or __ENV to avoid identical-cache distortion. Use k6 run --out json=results.json to archive runs for regression comparison. If you wire this into CI, fail the build on threshold breach—but cap VUs low (≤5) in CI to avoid burning quota.
Load testing LLM API with k6 is not about maximal requests per second; it is about understanding token economics under contention. The script above is ~80 lines, runs anywhere, and gives you the three numbers that predict user anger: first token wait, tokens per second, and error rate. Tune those, and the rest of your LLM stack stays honest.