Piping curl streaming output into jq for parsing

When you call a streaming LLM API from the command line, the raw bytes are a firehose of Server-Sent Events that are miserable to read. This guide shows how to build a robust curl jq parse streaming output pipeline that turns that firehose into clean, line-delimited JSON you can inspect, filter, or forward to another tool. We’ll work with the OpenAI streaming format because it’s the de facto standard for most gateways.

Step 1: Capture the raw stream with curl

The first mistake engineers make is letting curl buffer the response. Streaming endpoints send small chunks; if you omit -N, curl will wait for the connection to close before printing anything. Always use -N (or --no-buffer) and -sS to suppress the progress meter but keep errors.

curl -sN https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [{"role": "user", "content": "Say hello in 5 words."}]
  }'

If you’re hitting an OpenAI-compatible gateway such as n4n.ai, which fronts 240+ models behind one endpoint, the request shape is identical—only the base URL and auth header change. The raw output looks like this:

data: {"id":"chatcmpl-123","choices":[{"delta":{"role":"assistant","content":""},"index":0}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" there"},"index":0}]}

data: [DONE]

Note the blank lines between events. That’s SSE framing, not JSON.

Step 2: Strip the SSE framing before jq sees it

jq expects one JSON value per line (or a stream of JSON values). The data: prefix and the occasional carriage return will break it. Use sed to remove the prefix and tr to drop \r. Also filter out the [DONE] sentinel.

curl -sN https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Say hello in 5 words."}]}' \
  | tr -d '\r' \
  | sed 's/^data: //' \
  | grep -v '^\[DONE\]'

At this point your curl jq parse streaming output intermediate stage emits pure JSON lines:

{"id":"chatcmpl-123","choices":[{"delta":{"role":"assistant","content":""},"index":0}]}
{"id":"chatcmpl-123","choices":[{"delta":{"content":"Hello"},"index":0}]}
{"id":"chatcmpl-123","choices":[{"delta":{"content":" there"},"index":0}]}

Step 3: Pipe to jq for incremental parsing

Now invoke jq with -c (compact) and --unbuffered so it prints each result as soon as a line arrives. The delta objects have a content field only when there is text. Use select() to skip empty deltas and extract the string.

... | jq -c --unbuffered 'select(.choices[0].delta.content != null) | .choices[0].delta.content'

Output:

"Hello"
" there"

If you want the raw chunk JSON without extraction, just run jq -c '.'. The --unbuffered flag is critical: without it, jq may buffer multiple lines and defeat the streaming purpose. Your curl jq parse streaming output command should feel live—tokens appear as the model generates them.

Handling non-OpenAI streams

Some providers send NDJSON instead of SSE, or prefix with event:. Adjust the sed/grep accordingly. For NDJSON, skip Step 2 entirely and pipe directly to jq. For event: delta lines, strip those with sed '/^event:/d'.

Step 4: Reconstruct the full message or extract usage

Streaming is great for UX, but sometimes you need the assembled text or the final token count. Since each line is a separate JSON object, you can collect them with jq -s (slurp) at the end, but that waits for the stream to finish. Alternatively, use a small awk or paste to concatenate content strings.

To get the full reply text after the stream ends:

... | jq -c --unbuffered 'select(.choices[0].delta.content != null) | .choices[0].delta.content' \
  | jq -s -r 'join("")'

If the API includes a final usage chunk (OpenAI does not in the stream by default, but some gateways do), capture it:

... | jq -c --unbuffered 'select(.usage != null) | .usage'

For per-token metering—relevant when you run through a gateway that bills per token—you can sum choices[0].delta.content | length as a rough proxy, though proper counting requires a tokenizer.

Step 5: Verify the pipeline works without a network call

You don’t need a live API key to test the parsing logic. Fabricate an SSE stream with printf and run it through the exact same post-curl stages.

printf 'data: {"choices":[{"delta":{"content":"Hello"}}]}\r\n\r\ndata: {"choices":[{"delta":{"content":" world"}}]}\r\n\r\ndata: [DONE]\r\n' \
  | tr -d '\r' \
  | sed 's/^data: //' \
  | grep -v '^\[DONE\]' \
  | jq -c --unbuffered 'select(.choices[0].delta.content != null) | .choices[0].delta.content'

Expected output:

"Hello"
" world"

If you see that, the curl jq parse streaming output chain is correct; swap the printf for a real curl and you’re done. For a more thorough check, pipe the output to jq -e . to assert each line is valid JSON, or use jq empty which exits non-zero on parse error.

Step 6: Harden against errors and malformed chunks

Production streams drop. A provider may inject a non-JSON comment line or an error event. Wrap the jq filter in try to skip bad lines without killing the pipeline.

... | jq -c --unbuffered 'try (select(.choices[0].delta.content != null) | .choices[0].delta.content) catch empty'

Add curl -f (or --fail-with-body) so HTTP errors don’t produce a 200-style body that jq chokes on. If you’re behind a gateway with automatic fallback when a provider is degraded, you might still receive a stream from a secondary provider whose chunk shape differs slightly; the try guard keeps you resilient.

Also consider set -euo pipefail in scripts. A broken middle stage (e.g., sed receiving SIGPIPE) should propagate. With pipefail, a non-zero exit in curl surfaces instead of being masked by successful jq termination.

Step 7: Compose a reusable shell function

Don’t retype this every time. Put it in your .bashrc or a script.

stream_parse() {
  local url="$1"; local key="$2"; local body="$3"
  curl -sN -f "$url" \
    -H "Authorization: Bearer $key" \
    -H "Content-Type: application/json" \
    -d "$body" \
  | tr -d '\r' \
  | sed 's/^data: //' \
  | grep -v '^\[DONE\]' \
  | jq -c --unbuffered 'try (select(.choices[0].delta.content != null) | .choices[0].delta.content) catch empty'
}

Call it:

stream_parse https://api.openai.com/v1/chat/completions "$OPENAI_API_KEY" \
  '{"model":"gpt-4o-mini","stream":true,"messages":[{"role":"user","content":"Ping"}]}'

The curl jq parse streaming output pattern is now a one-liner you can embed in log tailing, tests, or quick model comparisons.

Step 8: Inspect raw chunks for debugging

When something looks off—missing tokens, weird latency—drop the jq filter and just look at the cleaned JSON lines.

... | sed 's/^data: //' | grep -v '^\[DONE\]' | jq -c '.'

Watch for finish_reason in the last chunk, or error objects mid-stream. Because jq is streaming, you’ll see them as they arrive. This beats saving a 50 MB log file and parsing post-hoc.

Why this matters for LLM tooling

CLI streaming inspection is not a toy. It’s how you debug prompt caching hints, verify that cache_control is honored, or confirm that a routing directive sent you to the model you asked for. A clean curl jq parse streaming output workflow turns an opaque binary-ish stream into actionable signals. Once you have line-delimited JSON, you can pipe into grep, awk, or a custom Go consumer just as easily.

Keep the pipeline unbuffered end-to-end. Any stage that waits for EOF will hide the exact moment a provider stalls—defeating the entire point of streaming.