Parsing streaming SSE output in a bash script with jq

Most LLM APIs stream completions as Server-Sent Events (SSE), and doing bash sse streaming jq parsing lets you pipe model output straight into Unix tools without standing up a service. This guide walks through a complete curl-to-jq pipeline that handles the data: prefix, the [DONE] sentinel, and partial JSON, so you can automate prompts from a shell. You’ll end with a script that prints streamed tokens and exits cleanly.

Step 1: Build the streaming curl request

Start with a plain curl call against an OpenAI-compatible chat completions endpoint. The critical flag is -N (or --no-buffer), which disables curl’s internal buffering so lines arrive as the server emits them. Without it, you’ll get one giant blob at the end.

ENDPOINT="https://api.openai.com/v1/chat/completions"
API_KEY="${OPENAI_API_KEY}"

curl -N "$ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role": "user", "content": "Say hello in three words."}]
  }'

If you hit an OpenAI-compatible gateway such as n4n.ai, a single /v1/chat/completions endpoint streams SSE for 240+ models and handles provider fallback, so your script only cares about the wire format. The request body stays identical.

Run that bare command first. You should see lines beginning with data: followed by JSON, ending with data: [DONE]. If you see nothing until the request finishes, double-check -N.

Step 2: Read the SSE stream line by line

Bash can’t parse a stream by itself; you need a read loop. Pipe curl into while IFS= read -r line. IFS= preserves leading whitespace, and -r stops backslash interpretation, which matters because JSON contains escaped characters.

curl -N "$ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Say hello in three words."}]}' \
| while IFS= read -r line; do
    printf 'RAW: %s\n' "$line"
  done

This is the skeleton of all bash sse streaming jq parsing: a tight loop where each line is inspected, stripped, and optionally handed to jq. Avoid for line in $(curl ...)—word splitting will mangle JSON.

Step 3: Strip the data: prefix and detect termination

SSE lines meant for the client start with data: . Some servers send comment lines starting with : as keep-alives. Filter inside the loop.

while IFS= read -r line; do
  # Skip empty lines and SSE comments
  [[ -z "$line" ]] && continue
  [[ "$line" == :* ]] && continue

  # Extract payload after "data: "
  if [[ "$line" == data:* ]]; then
    payload="${line#data: }"
    if [[ "$payload" == "[DONE]" ]]; then
      echo "STREAM ENDED"
      break
    fi
    printf 'JSON: %s\n' "$payload"
  fi
done

The ${line#data: } expansion removes the prefix without spawning sed. The [DONE] check must come before JSON parsing—jq will choke on that literal.

Step 4: Parse each chunk with jq

Each JSON payload looks like this:

{"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

The token lives at .choices[0].delta.content. Use jq -r for raw output and // empty to suppress null deltas (role markers often have empty deltas).

echo "$payload" | jq -r '.choices[0].delta.content // empty'

Integrated into the loop:

while IFS= read -r line; do
  [[ -z "$line" ]] && continue
  [[ "$line" == :* ]] && continue
  [[ "$line" != data:* ]] && continue

  payload="${line#data: }"
  [[ "$payload" == "[DONE]" ]] && break

  token=$(echo "$payload" | jq -r '.choices[0].delta.content // empty')
  [[ -n "$token" ]] && printf '%s' "$token"
done
printf '\n'

That printf '%s' prints tokens with no newline, mimicking typed output. The trailing printf '\n' adds one at the end.

Step 5: Accumulate or redirect the output

Printing to stdout is fine for demos, but real automation needs the full text. Append to a variable or a file.

full=""
while IFS= read -r line; do
  [[ "$line" == data:* ]] || continue
  payload="${line#data: }"
  [[ "$payload" == "[DONE]" ]] && break
  token=$(echo "$payload" | jq -r '.choices[0].delta.content // empty')
  full+="$token"
done

echo "ASSISTANT: $full" > response.txt

If you’re piping into another command, replace the echo with a FIFO or tee. Don’t accumulate huge strings in bash if you can stream to a file directly:

while IFS= read -r line; do
  [[ "$line" == data:* ]] || continue
  payload="${line#data: }"
  [[ "$payload" == "[DONE]" ]] && break
  echo "$payload" | jq -r '.choices[0].delta.content // empty' >> stream.out
done

Step 6: Handle errors and malformed events

SSE over flaky networks drops connections. Wrap curl with --retry 3 --retry-delay 1. Also, OpenAI-style APIs return HTTP 4xx with a JSON error body without stream mode if the initial request fails, but mid-stream errors can appear as a data: line containing an error object.

curl -N --retry 3 --retry-delay 1 "$ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Hi"}]}' \
| while IFS= read -r line; do
    [[ "$line" == data:* ]] || continue
    payload="${line#data: }"
    if [[ "$payload" == "[DONE]" ]]; then break; fi

    # Detect error object
    if echo "$payload" | jq -e '.error' >/dev/null 2>&1; then
      echo "API ERROR: $(echo "$payload" | jq -r '.error.message')" >&2
      exit 1
    fi

    echo "$payload" | jq -r '.choices[0].delta.content // empty'
  done

The jq -e '.error' returns a non-zero exit code when the key is absent, so the if only triggers on real errors. This is a robust pattern for bash sse streaming jq parsing in cron jobs.

Step 7: Verify the script end to end

Save the full loop as stream.sh, chmod +x, and run it against a known prompt.

#!/usr/bin/env bash
set -euo pipefail
ENDPOINT="${1:-https://api.openai.com/v1/chat/completions}"
API_KEY="${OPENAI_API_KEY}"

curl -N --retry 3 "$ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Count to three."}]}' \
| while IFS= read -r line; do
    [[ "$line" == data:* ]] || continue
    payload="${line#data: }"
    [[ "$payload" == "[DONE]" ]] && { echo; break; }
    echo "$payload" | jq -r '.choices[0].delta.content // empty' | tr -d '\n'
  done

Verification checklist:

• Tokens appear incrementally, not all at once (add sleep 0.01 after print to make it obvious).
• The script exits 0 and prints a final newline.
• Replacing stream:true with false and removing the loop yields identical final text (ignoring whitespace), confirming you parsed every delta.
• Kill the network mid-stream; curl retries or the loop exits with a non-zero code due to set -e on the subshell.

If those hold, your bash sse streaming jq parsing is production-ready for shell-based LLM automation. No Python, no Node, just curl and jq.