Out of order tokens streamed output is a subtle failure mode that shows up only under load or across provider failover. When your LLM response renders backwards, interleaves chunks, or drops sequence, you need a systematic way to trace where the stream broke. This guide walks through capturing raw frames, injecting sequence markers, and building a client buffer that makes the disorder visible and fixable.
Step 1: Capture raw stream frames without parsing
Never trust your application logs first. Capture the exact bytes from the LLM endpoint before your JSON parser, SSE decoder, or framework middleware touches them. Frameworks like LangChain or Vercel AI SDK wrap the stream and can silently drop or coalesce chunks, masking the real behavior.
Run a raw capture with curl and save to disk. Use --compressed only if you explicitly handle decompression; otherwise disable it to see raw bytes:
curl -N --no-compressed -X POST https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"count to 10"}]}' \
> raw_stream.txtThe -N disables curl’s own buffering. Open raw_stream.txt. You should see lines prefixed with data: terminated by double newlines. If any line arrives out of textual order, you have proof.
A minimal Python reader that prints chunk indices and content helps spot the break:
with open("raw_stream.txt") as f:
for i, line in enumerate(f):
if line.startswith("data:"):
print(i, line[5:].strip()[:80])If i increments but the semantic content jumps backwards—say token 5 appears before token 3—you have out of order tokens streamed output at the transport or source layer. Note the exact byte offsets; they become your regression fixtures.
Watch for hidden buffering
Some load balancers buffer SSE for 4–8 KB before flush. That does not reorder, but it batches, making manual inspection harder. Use tcpdump if you suspect intermediate proxies.
Step 2: Understand the protocol’s lack of built-in ordering
The OpenAI streaming schema sends choices[0].delta with no sequence number. Order is assumed to follow TCP. A typical frame:
{"id":"chatcmpl-1","choices":[{"delta":{"content":"Hello"},"index":0,"finish_reason":null}]}{"id":"chatcmpl-1","choices":[{"delta":{"content":" world"},"index":0,"finish_reason":null}]}There is no seq field. If a proxy fans out to multiple backends or retries a failed chunk, the client cannot detect duplication or reordering from the payload alone. This is why out of order tokens streamed output slips into production unnoticed until a user sees “world Hello”.
SSE vs NDJSON
Server-Sent Events mandate one event per data: line. NDJSON streams send raw JSON per line. Both preserve byte order on a single connection. Problems start when a gateway opens parallel connections and merges them, or when HTTP/2 stream reprioritization delays frames. Know which transport your provider uses before blaming the model.
Step 3: Inject synthetic sequence numbers at the edge
Place a thin shim between the provider and your app. It appends a monotonic counter to each chunk. This converts an implicit order into an explicit contract. Use a simple async generator in Python:
import json
import itertools
async def tag_stream(raw_stream):
counter = itertools.count()
async for chunk in raw_stream:
data = json.loads(chunk[5:])
data["_seq"] = next(counter)
yield f"data: {json.dumps(data)}\n\n"Now your client receives _seq alongside the delta. Any gap or decrease is measurable. Run this shim locally and point your app at it. You will immediately see whether the disorder originates upstream or in your own code.
Thread safety
If you proxy concurrently, use a lock or per-connection counter. A global counter across connections is fine for detection but will show gaps when two streams interleave—that is expected. Tag with stream_id plus _seq to avoid confusion.
Step 4: Distinguish transport reordering from provider failover
TCP delivers bytes in order on a single connection. Reordering requires either a broken proxy that merges streams, or an application-level fallback that replays a suffix. If you sit behind a gateway like n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback, a mid-stream provider switch can produce duplicate or reordered chunks when the secondary provider resumes the same id. That is a server-side event, not a bug in your socket.
To isolate, capture two traces:
- Direct to provider, bypassing the gateway.
- Through the gateway with identical requests.
Write a diff script that reports sequence gaps:
def seq_gaps(frames):
seqs = [f["_seq"] for f in frames]
return [(seqs[i], seqs[i+1]) for i in range(len(seqs)-1) if seqs[i+1] != seqs[i]+1]
print("gaps direct:", seq_gaps(direct_frames))
print("gaps gateway:", seq_gaps(gateway_frames))If gaps appear only via the gateway, the fallback path is the culprit. If both show gaps, the provider itself is emitting out of order tokens streamed output (rare but observed under overloaded GPUs). File a ticket with the raw capture attached.
HTTP/2 caveats
Multiplexed streams on one connection can be reordered by the client stack if you read them concurrently. Ensure your client reads the response body serially.
Step 5: Build a client-side reordering buffer
Assume you cannot fix the upstream immediately. Buffer chunks and emit them in sequence order once you have a contiguous window. A TypeScript example for browser fetch:
async function streamInOrder(res: Response, onToken: (s: string) => void) {
const reader = res.body!.getReader();
const decoder = new TextDecoder();
const buffer = new Map<number, string>();
let nextExpected = 0;
let stallStart = 0;
for await (const chunk of readSSE(reader, decoder)) {
const { _seq, delta } = JSON.parse(chunk);
buffer.set(_seq, delta?.content ?? "");
if (stallStart === 0 && !buffer.has(nextExpected)) stallStart = Date.now();
while (buffer.has(nextExpected)) {
onToken(buffer.get(nextExpected)!);
buffer.delete(nextExpected);
nextExpected++;
stallStart = 0;
}
if (stallStart && Date.now() - stallStart > 2000) {
// flush out-of-order to avoid hang
const keys = [...buffer.keys()].sort((a,b)=>a-b);
for (const k of keys) { onToken(buffer.get(k)!); buffer.delete(k); nextExpected = k+1; }
stallStart = 0;
}
}
while (buffer.has(nextExpected)) {
onToken(buffer.get(nextExpected)!);
buffer.delete(nextExpected);
nextExpected++;
}
}This trades latency for correctness: a missing chunk stalls output until timeout. Set a max stall (e.g., 2s) then flush out-of-order to avoid hangs.
Backpressure
If you buffer too aggressively, memory grows during a stall. Cap the map size at 500 entries; beyond that, drop and report. The goal is to survive transient reordering, not to build a durable queue.
Step 6: Add end-to-end assertions in CI
Capture a known-good and a known-bad stream during chaos testing. Replay them through your buffer in unit tests:
def test_buffer_reorders():
frames = [{"_seq": 0, "delta": {"content": "A"}},
{"_seq": 2, "delta": {"content": "C"}},
{"_seq": 1, "delta": {"content": "B"}}]
out = run_buffer(frames)
assert out == "ABC", out
def test_buffer_stall_flush():
frames = [{"_seq": 0, "delta": {"content": "A"}},
{"_seq": 5, "delta": {"content": "F"}}]
out = run_buffer(frames, max_stall=0.01)
assert out == "AF", outRun this against recorded production anomalies. If the test fails, your detection is still blind to a reorder pattern. Use property-based testing (Hypothesis) to generate random seq permutations and assert the buffer always produces the sorted concatenation.
Step 7: Verify success
Success means three things: (1) raw captures show monotonic _seq with zero gaps in steady state; (2) your client buffer flushes tokens in the exact order the model generated them, confirmed by replaying a labeled transcript; (3) production dashboards report zero “stall timeout” events for at least a week.
To verify, replay the raw_stream.txt from Step 1 through the tagged shim and buffer. Diff the rendered text against the provider’s non-streamed completion. They should match character-for-character. If they do, you have killed out of order tokens streamed output in your pipeline.
Monitoring
Export a counter stream_seq_gaps_total and stream_stall_flushes_total to Prometheus. Alert if either rises above baseline. That turns a silent corruption into a pagable metric.