Most LLM streaming endpoints speak Server-Sent Events over HTTP, and a naive line reader will break on partial frames or large payloads. Using go bufio.scanner sse parsing gives you a buffered, line-oriented approach that handles chunked transfers without pulling the whole response into memory, which matters when you’re processing token streams that can run for minutes.
Step 1: Understand the SSE wire format
Server-Sent Events are not JSON blobs separated by commas. They are a line-based protocol defined by the WHATWG HTML standard. Each event is a sequence of fields, one per line, where a line is terminated by LF, CR, or CRLF. Fields are data:, event:, id:, and retry:. A blank line signals the end of an event.
Example stream:
data: {"token":"hello"}
data: {"token":" world"}
event: ping
data: {}The data lines for a single event are joined with LF if there are multiple. The event field sets the event type; if absent, it defaults to message. When you do go bufio.scanner sse parsing, your job is to buffer lines until you hit a blank line, then emit the assembled event.
Step 2: Open a streaming HTTP request
You need a client that does not buffer the response body. The standard net/http client already streams the body, but you must set the right headers. For an OpenAI-compatible chat completion stream, pass "stream": true in the JSON body and set Accept: text/event-stream.
If you’re consuming a gateway like n4n.ai, the OpenAI-compatible /v1/chat/completions endpoint accepts that same shape and returns SSE frames with per-token usage metadata. The code below is generic:
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "text/event-stream")
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status %d", resp.StatusCode)
}Do not use ioutil.ReadAll. You want the stream.
Step 3: Configure bufio.Scanner for line reading
A bufio.Scanner splits input into tokens. Its default split function ScanLines is exactly what we need for SSE because it strips line endings. However, the default maximum token size is 64KB. LLM responses can contain a single data: line with a large JSON payload (e.g., a full completion or a tool call), so you must grow the buffer.
scanner := bufio.NewScanner(resp.Body)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)This allocates an initial 64KB and allows growth up to 1MB per line. If you expect larger frames, raise the max. The core go bufio.scanner sse parsing loop reads lines until scanner.Scan() returns false.
Step 4: Assemble events from lines
We accumulate fields in a struct. On a blank line, if we have collected any data, we dispatch the event and reset.
type SSEEvent struct {
Event string
Data string
ID string
}
func parseSSE(scanner *bufio.Scanner, out chan<- SSEEvent) error {
var cur SSEEvent
for scanner.Scan() {
line := scanner.Text()
if line == "" {
if cur.Data != "" || cur.Event != "" {
if cur.Event == "" {
cur.Event = "message"
}
out <- cur
}
cur = SSEEvent{}
continue
}
if strings.HasPrefix(line, "data:") {
val := strings.TrimPrefix(line, "data:")
val = strings.TrimPrefix(val, " ")
if cur.Data != "" {
cur.Data += "\n" + val
} else {
cur.Data = val
}
} else if strings.HasPrefix(line, "event:") {
cur.Event = strings.TrimSpace(strings.TrimPrefix(line, "event:"))
} else if strings.HasPrefix(line, "id:") {
cur.ID = strings.TrimSpace(strings.TrimPrefix(line, "id:"))
}
// ignore retry and unknown fields
}
return scanner.Err()
}Note the handling of the leading space after the colon. The spec allows exactly one optional space; we strip it. Multi-line data is joined with \n, matching browser behavior. Also skip lines starting with : (comment/keep-alive) before the prefix checks if you want strict compliance.
Step 5: Run the parser in a goroutine and dispatch over a channel
The topic cluster here is Go streaming with goroutines and channels. The blocking Scan call should not sit on your main goroutine if you want to process events concurrently or cancel cleanly. Spin up a goroutine, pass it a channel, and close the channel when the stream ends.
events := make(chan SSEEvent, 16)
go func() {
defer close(events)
if err := parseSSE(scanner, events); err != nil {
log.Printf("scanner error: %v", err)
}
}()
for ev := range events {
switch ev.Event {
case "message":
var chunk map[string]any
if err := json.Unmarshal([]byte(ev.Data), &chunk); err != nil {
log.Printf("bad json: %v", err)
continue
}
// handle chunk
case "ping":
// ignore or reset timeout
}
}This design decouples network I/O from business logic. You can add a context.Context to terminate the goroutine by closing the response body or using a canceled context in the request.
Step 6: Handle partial reads, timeouts, and reconnection
SSE connections can drop. scanner.Err() may return io.ErrUnexpectedEOF or a net.OpError on timeout. Wrap the stream consumer in a retry loop that respects the retry: field if present, or a fixed backoff.
backoff := time.Second
for attempt := 0; attempt < 5; attempt++ {
if err := streamOnce(ctx, endpoint, body, apiKey); err != nil {
log.Printf("stream ended: %v; retrying in %v", err, backoff)
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
backoff *= 2
continue
}
break
}Also set a http.Transport with IdleConnTimeout and ResponseHeaderTimeout if you see stalls. The go bufio.scanner sse parsing layer doesn’t care about reconnection; that’s the caller’s job.
Step 7: Verify your parser against a live or local stream
You need proof it works. The fastest verification is a tiny local SSE server and your client in main. Here is a minimal server:
http.HandleFunc("/stream", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
flusher := w.(http.Flusher)
for i := 0; i < 3; i++ {
fmt.Fprintf(w, "data: {\"index\":%d}\n\n", i)
flusher.Flush()
time.Sleep(100 * time.Millisecond)
}
})Point your client at http://localhost:8080/stream. Success criteria: your program prints three parsed JSON objects with index 0,1,2 and exits cleanly without scanner errors. If you point it at a real LLM endpoint, you should see data: lines that decode to OpenAI-style chunk objects with choices[0].delta.content.
If you used the n4n.ai endpoint mentioned earlier, a successful run will emit per-token data: frames and a final frame containing usage metadata, confirming that your go bufio.scanner sse parsing handles both mid-stream and terminal events.
Common pitfalls
- Forgetting to strip the space after
data:. The spec saysdata: xyieldsx, notx. - Assuming every line is JSON. SSE allows comments starting with
:(colon) for keep-alive. Skip lines that start with:. - Blocking the channel. If your consumer is slower than the network, an unbuffered channel will stall the scanner goroutine and may cause the TCP window to fill. Use a buffered channel or process asynchronously.
- Ignoring
scanner.Err(). A dropped connection returns an error afterScanreturns false. Always check it.
Wrapping up
Robust go bufio.scanner sse parsing is mostly discipline: respect the line protocol, grow the scanner buffer, assemble events on blank lines, and push them over a channel from a dedicated goroutine. That pattern scales from a simple CLI to a production gateway consumer processing hundreds of concurrent streams.