Streaming vs non-streaming load testing for LLM endpoints demands different assumptions about connection lifetime, token throughput, and client behavior. Treat them as identical and you’ll either saturate your proxy with idle connections or miss latency regressions that only show up after the first token. This guide lays out an ordered path to test both modes realistically.
1. Define the metrics that matter
Non-streaming calls are simple: you measure requests per second (RPS), error rate, and end-to-end latency. The client opens a connection, waits, and gets a complete JSON payload.
Streaming changes the shape. You must track time-to-first-token (TTFT), inter-token latency (ITL), and total generation time separately. A request that returns 2,000 tokens over 40 seconds holds a connection open far longer than a 200-token non-streaming reply that finishes in 3 seconds.
Pitfall: aggregating only RPS hides the fact that streaming workloads keep sockets busy. Under the same RPS, streaming can require 5–10x the concurrent connections.
2. Choose a client that parses Server-Sent Events
Most load tools assume a single response body. For streaming LLM APIs, you need one that consumes chunked HTTP and reads SSE frames. k6, Locust, and raw Python aiohttp all work; avoid tools that buffer the whole response.
Below is a minimal k6 script for a non-streaming call to an OpenAI-compatible endpoint:
import http from 'k6/http';
import { check } from 'k6';
export const options = { vus: 50, duration: '2m' };
export default function () {
const res = http.post('https://api.example.com/v1/chat/completions',
JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Summarize this log' }],
stream: false
}),
{ headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + __ENV.API_KEY } }
);
check(res, { 'status 200': (r) => r.status === 200 });
}For streaming, you must read the response stream and parse data: lines. Note that accurate streaming vs non-streaming load testing requires a client that yields between chunks rather than buffering:
import http from 'k6/http';
import { check } from 'k6';
export const options = { vus: 50, duration: '2m' };
export default function () {
const res = http.post('https://api.example.com/v1/chat/completions',
JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Write a long story' }],
stream: true
}),
{ headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + __ENV.API_KEY } }
);
let firstTokenSeen = false;
let ttft = 0;
const start = Date.now();
res.body.split('\n').forEach((line) => {
if (line.startsWith('data:') && !firstTokenSeen) {
firstTokenSeen = true;
ttft = Date.now() - start;
}
});
check(res, { 'status 200': (r) => r.status === 200 });
}k6’s http.post buffers the full body by default. For true streaming behavior, use a Node.js script with fetch and ReadableStream, or a Python aiohttp client:
import aiohttp, asyncio, time
async def stream_req(session, prompt):
start = time.monotonic()
async with session.post(
"https://api.example.com/v1/chat/completions",
json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": prompt}], "stream": True},
headers={"Authorization": "Bearer " + API_KEY}
) as resp:
ttft = None
async for line in resp.content:
if line.startswith(b"data:"):
if ttft is None:
ttft = time.monotonic() - start
return ttft
async def main():
async with aiohttp.ClientSession() as s:
await asyncio.gather(*[stream_req(s, "hello") for _ in range(100)])
asyncio.run(main())3. Model a realistic traffic mix
Production LLM apps rarely are 100% streaming or 100% batch. A typical mix is 70% streaming chat interactions and 30% non-streaming background jobs (embeddings, summarization, eval runs). Your load test must reflect that.
Use scenario weighting in your executor:
export const options = {
scenarios: {
stream_chat: { executor: 'constant-vus', vus: 70, duration: '5m' },
batch_jobs: { executor: 'constant-vus', vus: 30, duration: '5m' }
}
};If you route through a gateway like n4n.ai, its OpenAI-compatible endpoint addresses 240+ models with automatic fallback when a provider is degraded. Pin a specific model in tests unless you explicitly want to exercise fallback paths—otherwise variance from fallback will muddy your results.
4. Measure token throughput, not just request rate
A request is a poor unit of work for LLMs. One streaming request might emit 4,000 tokens; another might emit 20. Track tokens per second at the client and compare against provider limits.
Add token counting to your script by summing usage.completion_tokens (non-streaming) or counting SSE deltas (streaming). Gateways with per-token usage metering let you cross-check client counts against billed usage—useful for catching silent truncation.
Tradeoff: counting tokens in the client adds CPU overhead to the load generator. Run generators on machines separate from the service under test.
5. Tune connection and OS limits
Streaming holds connections open for the full generation window. If your average TTFT is 500ms and generation lasts 30s, a target of 100 RPS needs ~3,000 concurrent sockets. Default Linux ephemeral port range and ulimit -n will starve the test.
Raise limits before running:
ulimit -n 65536
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w net.core.somaxconn=4096Also set HTTP keep-alive on the client. Non-streaming can reuse connections aggressively; streaming benefits less because each connection is busy for seconds.
Pitfall: load generators behind NAT or a single IP may hit provider per-IP rate limits. Distribute generators across subnets or use a gateway that honors client routing directives to spread load.
6. Analyze results with the right lenses
For non-streaming, p95 latency and error rate are sufficient. For streaming, plot TTFT and ITL distributions separately. A degraded provider may keep TTFT low (quick acknowledgment) but slow ITL to a crawl—your users see a stalling cursor.
Common pitfalls in streaming vs non-streaming load testing:
- Counting a 200 status as success when the stream aborts mid-way. Check for the
[DONE]sentinel. - Ignoring client buffer bloat. If your test client buffers tokens, you measure network, not user-perceived latency.
- Mixing model sizes without labeling. A 70B model streams slower than a 7B; aggregate metrics blur this.
Example check for stream completion:
async def stream_req(session, prompt):
done = False
async with session.post(...) as resp:
async for line in resp.content:
if b"[DONE]" in line:
done = True
return done7. Automate with guardrails in CI
Don’t run a one-off siege. Put the test in CI with thresholds that fail the build:
export const options = {
thresholds: {
'http_req_duration': ['p95<3000'],
'stream_ttft': ['p95<1500']
}
};For streaming, emit a custom metric:
import { Trend } from 'k6/metrics';
const ttftTrend = new Trend('stream_ttft');
// inside stream parse: ttftTrend.add(ttft);Run nightly against a staging endpoint. Keep payloads representative—use production-like prompts, not “hi”. A 10-token prompt stresses TTFT; a 2,000-token prompt stresses token throughput and memory.
8. Common tradeoffs summary
- Fidelity vs simplicity: Real SSE parsing is harder but the only way to get TTFT.
- Cost: Streaming tests burn more tokens because generations run to completion. Cap test duration.
- Gateway caching: If your gateway forwards provider cache-control hints, repeated prompts may hit cache and skew latency. Vary prompts or disable cache in tests.
Streaming vs non-streaming load testing is not a checkbox. Build the mix, measure tokens, tune sockets, and watch the stream sentinel. Do that and your capacity plan will survive contact with real traffic.