Long LLM completions over Server-Sent Events fail when a proxy times out or the client switches networks. Implementing sse reconnection last-event-id llm logic lets you resume a stream from the last delivered token instead of restarting the request and paying for duplicated generation. This how-to walks through the server and client changes required to make that work against an OpenAI-compatible streaming endpoint.
Why naive retries waste tokens
A naive client that catches a fetch error and re-issues the chat completion request will get a fresh stream starting from token zero. For a 2,000-token completion that dies at token 1,800, you just burned 1,800 tokens of duplicate compute and your UI shows a jarring reset. The SSE protocol already defines a clean recovery mechanism: every event can carry an id, and the client can send Last-Event-ID on reconnect. The hard part is that most LLM providers are stateless—they don’t buffer your in-flight completion—so resumption requires either a stateful proxy or client-side checkpointing of delivered text.
Step 1: Emit proper SSE frames with id and retry
Your streaming endpoint must assign a monotonically increasing id to each chunk and send the retry: directive once at the start. The OpenAI streaming format wraps content in data: lines; you keep that but prefix an id:.
# FastAPI-style generator (illustrative SSE framing)
async def stream_completion(request):
yield "retry: 2000\n\n"
for i, chunk in enumerate(generate_tokens(request.prompt)):
# chunk is raw token text
payload = json.dumps({"choices": [{"delta": {"content": chunk}}]})
yield f"id: {i}\n"
yield f"data: {payload}\n\n"The id must be a string the server can later map to an offset. For LLM streams, the chunk index is sufficient if the server buffers the sequence. If you front requests with a gateway, ensure it does not strip the id field—some middleboxes rewrite SSE.
Step 2: Parse the stream and persist the last ID
On the client, use fetch with a streaming body reader. EventSource does not allow custom headers, so you cannot set Last-Event-ID manually; you must parse SSE yourself or use a polyfill that supports the header. Below is a minimal parser that records the latest ID.
let lastEventId = null;
const textDecoder = new TextDecoder();
async function readSSE(reader, onToken) {
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += textDecoder.decode(value, { stream: true });
const events = buffer.split("\n\n");
buffer = events.pop();
for (const evt of events) {
const lines = evt.split("\n");
let id, data = "";
for (const line of lines) {
if (line.startsWith("id: ")) id = line.slice(4);
else if (line.startsWith("data: ")) data += line.slice(6);
}
if (id) lastEventId = id;
if (data) {
const json = JSON.parse(data);
const token = json.choices?.[0]?.delta?.content;
if (token) onToken(token);
}
}
}
}Persist lastEventId to memory (or sessionStorage for tab restores). This is the core of sse reconnection last-event-id llm resilience: without storing the ID, you have nothing to send back.
Step 3: Reconnect using the Last-Event-ID header
When the stream breaks, open a new request and pass the saved ID in the Last-Event-ID header. The server should seek to that offset and emit only subsequent chunks.
async function connect(prompt, onToken) {
const headers = {
"Content-Type": "application/json",
Authorization: `Bearer ${API_KEY}`,
};
if (lastEventId) headers["Last-Event-ID"] = lastEventId;
const res = await fetch("/v1/chat/completions", {
method: "POST",
headers,
body: JSON.stringify({
model: "gpt-4o-mini",
messages: prompt,
stream: true,
}),
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const reader = res.body.getReader();
await readSSE(reader, onToken);
}Wrap this in a reconnect loop with exponential backoff. Start at 500 ms, double up to 8 s, and add jitter.
async function withReconnect(prompt, onToken) {
let attempt = 0;
while (true) {
try {
await connect(prompt, onToken);
return; // stream completed cleanly
} catch (err) {
attempt++;
const delay = Math.min(8000, 500 * 2 ** attempt) + Math.random() * 300;
await new Promise((r) => setTimeout(r, delay));
}
}
}Browser limitation with EventSource
Native EventSource automatically sends Last-Event-ID on reconnect, but only if the browser initiated the reconnect itself after a network drop—and it cannot send the Authorization header or a JSON body. For LLM APIs you need POST with a bearer token, so fetch plus manual parsing is the pragmatic path.
Step 4: Make the server replay from the offset
Stateless LLM endpoints will ignore Last-Event-ID and regenerate from scratch. You need a stateful layer that buffers the completion keyed by a stream ID (not just the chunk index, because the same index may appear across different requests). Two patterns work:
- Server-side buffer: The proxy assigns a
stream_id, mapsLast-Event-IDto an in-memory ring of sent chunks, and replays missing ones before continuing live generation. - Client-side checkpoint: The client sends the full prompt plus the already-received text as a
prefixparameter (if the model supports prompt continuation). This avoids server state but costs input tokens for the prefix.
If you front your models with an OpenAI-compatible gateway such as n4n.ai, confirm it buffers the completion and respects Last-Event-ID; it already honors client routing directives and forwards provider cache-control hints, but replay is a separate concern you must validate. Without buffer support, sse reconnection last-event-id llm handling degrades to a full restart.
Step 5: Verify with a forced disconnect
You cannot claim reconnection works until you break the pipe on purpose. Use a local proxy that drops the connection after N bytes, or simply kill your Wi-Fi mid-stream in a test harness.
A minimal Node verification script:
let received = [];
let firstId = null;
await withReconnect(
[{ role: "user", content: "Write a 500-word essay." }],
(tok) => {
received.push(tok);
if (!firstId) firstId = lastEventId;
}
);
// After forced disconnect at ~30%:
console.log("Chunks after reconnect start:", received.length);
console.log("Last ID at drop:", firstId);Success criteria:
- The concatenated string after recovery is a strict suffix of the full completion (no repeated sentences).
lastEventIdadvances monotonically; the reconnect request includes the header (inspect DevTools network tab).- Token usage from your metering (or provider bill) shows one generation, not two. If you see duplicate input+output tokens, the server ignored the offset.
For a Python client, the same logic applies with requests streaming:
import json, requests
last_id = None
def stream():
global last_id
headers = {"Authorization": f"Bearer {KEY}"}
if last_id:
headers["Last-Event-ID"] = last_id
with requests.post(URL, json=payload, headers=headers, stream=True) as r:
for line in r.iter_lines():
if line.startswith(b"id: "):
last_id = line[4:].decode()
elif line.startswith(b"data: "):
data = json.loads(line[6:])
yield data["choices"][0]["delta"].get("content", "")Run this in a loop, unplug the network after a few seconds, and assert the yielded text has no overlap with a pre-disconnect snapshot.
Edge cases that bite in production
Multiline data: fields are legal; your parser must accumulate until a blank line, not split on every \n. The retry: value is a hint—clients can ignore it. If your gateway does automatic fallback when a provider is degraded, a reconnect may hit a different upstream model; pin the model name and stream ID so the replay buffer stays valid. Finally, set a max reconnect age: if lastEventId is older than your buffer TTL, fail loud rather than silently regenerating.
Solid sse reconnection last-event-id llm support is not free, but the token savings on long outputs and the UX win of uninterrupted text make it worth the stateful proxy or checkpointing work.