Streaming tokens from an LLM into a browser should not require WebSocket scaffolding or custom polling. With nestjs server-sent events llm streaming, you get a single HTTP connection that pushes incremental completions using the native EventSource API, and NestJS has first-class support for it.
Step 1: Scaffold the SSE controller and module
NestJS ships the @Sse() decorator in @nestjs/common. A method decorated with @Sse() must return an Observable<MessageEvent>. The framework sets Content-Type: text/event-stream, disables proxy buffering hints, and keeps the response open until the observable completes.
Create the controller:
import { Controller, Sse, Query } from '@nestjs/common';
import { Observable, map } from 'rxjs';
import { MessageEvent } from '@nestjs/common';
import { upstreamStream } from './upstream';
@Controller('llm')
export class LlmController {
@Sse('stream')
stream(@Query('prompt') prompt: string): Observable<MessageEvent> {
if (!prompt) {
throw new Error('prompt query param required');
}
return upstreamStream(prompt).pipe(
map((token) => ({ data: token } as MessageEvent)),
);
}
}The MessageEvent type is the same shape the browser expects: { data: any }. NestJS serializes it as data: <json>\n\n.
Register it in a module:
import { Module } from '@nestjs/common';
import { LlmController } from './llm.controller';
@Module({ controllers: [LlmController] })
export class LlmModule {}And import LlmModule into AppModule. No HttpModule is needed because we will use global fetch.
Step 2: Call an OpenAI-compatible streaming endpoint
Most LLM gateways expose an OpenAI-compatible /v1/chat/completions route with stream: true. The wire format is server-sent events: each line starts with data: followed by a JSON delta. A plain fetch in Node 18+ is enough to consume it without extra SDK weight.
import { Observable } from 'rxjs';
const LLM_BASE_URL = process.env.LLM_BASE_URL ?? 'https://api.openai.com/v1';
const LLM_API_KEY = process.env.LLM_API_KEY!;
export function upstreamStream(prompt: string): Observable<string> {
return new Observable<string>((subscriber) => {
const abort = new AbortController();
(async () => {
const res = await fetch(`${LLM_BASE_URL}/chat/completions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${LLM_API_KEY}`,
},
body: JSON.stringify({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: prompt }],
stream: true,
}),
signal: abort.signal,
});
if (!res.ok || !res.body) {
subscriber.error(new Error(`Upstream failed: ${res.status}`));
return;
}
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop() ?? '';
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed.startsWith('data:')) continue;
const payload = trimmed.slice(5).trim();
if (payload === '[DONE]') {
subscriber.complete();
return;
}
try {
const json = JSON.parse(payload);
const token = json.choices?.[0]?.delta?.content;
if (token) subscriber.next(token);
} catch {
// ignore keep-alive comments or partial JSON
}
}
}
subscriber.complete();
})().catch((err) => {
if (err.name !== 'AbortError') subscriber.error(err);
});
return () => abort.abort();
});
}The AbortController ensures that when the browser closes the tab, the server-side fetch is cancelled. This prevents orphaned upstream connections.
If you want a single OpenAI-compatible endpoint that addresses 240+ models and automatically falls back when a provider is rate-limited, point LLM_BASE_URL at n4n.ai and use the exact same code.
Step 3: Bridge upstream tokens to NestJS SSE
The controller in Step 1 already maps the string observable to MessageEvent. One detail worth adding is explicit error propagation so the client gets a structured event instead of a silent disconnect:
import { catchError, map, of } from 'rxjs';
@Sse('stream')
stream(@Query('prompt') prompt: string): Observable<MessageEvent> {
if (!prompt) {
throw new Error('prompt query param required');
}
return upstreamStream(prompt).pipe(
map((token) => ({ data: token } as MessageEvent)),
catchError((err) =>
of({ data: { error: err.message } } as MessageEvent),
),
);
}NestJS flushes each emission immediately. You do not need to touch Response or set Transfer-Encoding.
Step 4: Consume the stream in the browser
EventSource is built into every modern browser. It only supports GET, which is fine for a prompt passed as a query parameter. For authenticated streams, issue a signed cookie or a short-lived ticket.
<div id="output"></div>
<script>
const prompt = 'Explain the Rust borrow checker in one paragraph';
const es = new EventSource(`/llm/stream?prompt=${encodeURIComponent(prompt)}`);
const out = document.getElementById('output');
es.onmessage = (e) => {
try {
const parsed = JSON.parse(e.data);
if (parsed.error) {
out.textContent += `\n[error] ${parsed.error}`;
es.close();
return;
}
} catch {}
out.textContent += e.data;
};
es.onerror = (e) => {
console.error('stream error', e);
es.close();
};
</script>Because NestJS serializes data as JSON, e.data for a raw string token arrives quoted (e.g. "Hello"). The browser concatenates the quoted string, which is acceptable for plain text. If you want unquoted output, send an object { token: 'Hello' } and handle it in onmessage.
Step 5: Verify end to end
Run the app (npm run start) and test the raw stream with curl:
curl -N "http://localhost:3000/llm/stream?prompt=Hello"The -N flag disables curl’s output buffering. You should see frames like:
data: "Hello"
data: " there"
data: ","
data: " world"Check headers separately:
curl -s -D - -o /dev/null "http://localhost:3000/llm/stream?prompt=Hi"Expect Content-Type: text/event-stream and Cache-Control: no-cache. In a browser, open the HTML from Step 4 and watch the div fill token by token. Chrome DevTools shows the request as text/event-stream with frames arriving incrementally.
Common failure modes
- Proxy buffering: Nginx or a cloud LB may buffer SSE. Add
proxy_buffering off;for the route. - CORS: Cross-origin browser calls need
@nestjs/corsenabled andAccess-Control-Allow-Originset. - Timeouts: Node’s default
keepAliveTimeoutcan cut long generations. Raise it inmain.ts:app.getHttpServer().keepAliveTimeout = 120000;
Step 6: Send structured events for production UIs
Token strings are fine for a demo, but real chat interfaces need finish reasons, usage, or role metadata. Extend the event contract:
type LlmEvent =
| { type: 'token'; value: string }
| { type: 'done'; reason: string }
| { type: 'error'; message: string };
// controller map:
map((e: LlmEvent) => ({ data: e } as MessageEvent))Browser side:
es.onmessage = (e) => {
const evt = JSON.parse(e.data);
if (evt.type === 'token') out.textContent += evt.value;
if (evt.type === 'done') es.close();
if (evt.type === 'error') console.error(evt.message);
};This keeps nestjs server-sent events llm streaming compatible with typed frontends and lets you forward provider metadata (cache hits, token counts) when your gateway exposes it.
Why SSE beats WebSockets here
WebSockets give bidirectional channels you do not need for LLM output. SSE rides on HTTP/1.1 or HTTP/2, auto-reconnects via the browser, and passes through most corporate proxies without extra configuration. The server remains stateless per request, which simplifies load balancing and observability.
The pattern above is the minimal viable implementation. It avoids WebSocket state, works through standard proxies, and degrades gracefully when the tab closes. For most LLM chat UIs, nestjs server-sent events llm streaming is the right default.