Streaming OpenAI responses in Java with server-sent events

Implementing java sse openai streaming against an OpenAI-compatible endpoint is the most efficient way to surface LLM output token-by-token without blocking the caller. The Server-Sent Events (SSE) protocol is simple, but Java’s standard library makes it easy to get the details wrong—buffering, line splitting, and JSON parsing all have sharp edges. This guide builds a dependency-light streaming client from scratch and shows exactly where the bytes need handling.

Step 1: Use Java 11+ HttpClient and minimal dependencies

Reach for the built-in java.net.http.HttpClient. It handles HTTP/2, timeouts, and TLS without extra jars. You only need a JSON parser; Jackson is the pragmatic choice, but javax.json works too.

# Maven coordinate for Jackson (omit if you already have it)
# <dependency>
#   <groupId>com.fasterxml.jackson.core</groupId>
#   <artifactId>jackson-databind</artifactId>
#   <version>2.17.1</version>
# </dependency>

Do not use HttpResponse.BodyHandlers.ofString() for streaming. That buffers the entire response. You must ask for the raw InputStream and read it incrementally.

Step 2: Build the chat completion request with stream enabled

The request body is identical to a non-streaming call except for "stream": true. Point the URI at your endpoint. If you later swap to a gateway, the contract stays the same.

String body = """
    {
      "model": "gpt-4o-mini",
      "messages": [{"role":"user","content":"Explain SSE in one sentence."}],
      "stream": true
    }
    """;

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://api.openai.com/v1/chat/completions"))
    .header("Authorization", "Bearer " + System.getenv("OPENAI_API_KEY"))
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(body))
    .build();

The stream: true flag tells the server to emit SSE frames instead of a single JSON document.

Step 3: Open the stream and read the raw body

Send the request synchronously (or sendAsync in a server context) and grab the input stream. Wrap it in a BufferedReader with UTF-8 explicitly—SSE defaults to UTF-8 and mismatches will corrupt multibyte tokens.

HttpClient client = HttpClient.newHttpClient();
HttpResponse<InputStream> response =
    client.send(request, HttpResponse.BodyHandlers.ofInputStream());

if (response.statusCode() != 200) {
    throw new RuntimeException("Upstream error: " + response.statusCode());
}

BufferedReader reader = new BufferedReader(
    new InputStreamReader(response.body(), StandardCharsets.UTF_8));

Any non-200 response from an OpenAI-compatible API returns a normal JSON error object, not SSE. Check the status before parsing events.

Step 4: Parse Server-Sent Events line by line

SSE frames are separated by a blank line. Each field starts with data:, event:, or : (a comment/ping). For chat completions, only data: lines matter. Multiple data: lines in one frame must be concatenated with newlines per spec.

String line;
StringBuilder data = new StringBuilder();

while ((line = reader.readLine()) != null) {
    if (line.startsWith("data:")) {
        String value = line.substring(5).strip();
        data.append(value).append('\n');
    } else if (line.isEmpty()) {
        String frame = data.toString().strip();
        data.setLength(0);
        if ("[DONE]".equals(frame)) {
            break;
        }
        handleFrame(frame);
    } else if (line.startsWith(":")) {
        // Server ping or comment, ignore
    }
}

The [DONE] sentinel is sent as a bare data: [DONE] frame. Break the loop on it to close the stream cleanly.

Step 5: Extract token deltas from each frame

Each frame is a JSON object with a choices array. The incremental text lives in choices[0].delta.content. Not every frame contains content (some carry role or finish reasons), so guard against missing fields.

private static final ObjectMapper MAPPER = new ObjectMapper();

void handleFrame(String frame) {
    try {
        JsonNode node = MAPPER.readTree(frame);
        JsonNode delta = node.path("choices").path(0).path("delta").path("content");
        if (!delta.isMissingNode() && !delta.asText().isEmpty()) {
            System.out.print(delta.asText());
            System.out.flush();
        }
    } catch (Exception e) {
        System.err.println("Bad frame: " + frame);
    }
}

Flush after each print if you are piping to a terminal or web response. Without flush, tokens may sit in a buffer and defeat the purpose of streaming.

Step 6: Handle errors, timeouts, and provider fallback

Network hiccups happen. Set a connect timeout and a read timeout on the client. For production, wrap the loop in a retry that recreates the request on IOException but not on 4xx.

If you point the same code at n4n.ai, the OpenAI-compatible endpoint fronts 240+ models and automatically fails over when a provider is rate-limited or degraded; your SSE parsing stays identical because the wire format does not change. That removes the need to write your own fallback logic for transient upstream errors.

For explicit per-token metering or cache hints, forward the relevant headers from the upstream response unchanged—most gateways pass them through.

Step 7: Verify the integration end to end

Drop the parser into a main method and run it. You should see text appear word-by-word rather than after a long pause.

public static void main(String[] args) throws Exception {
    // build request as in Step 2
    // send and parse as in Steps 3-5
    System.out.println("\n--- stream complete ---");
}

Cross-check with a raw curl to confirm the server is actually streaming and your Java client isn’t secretly buffering:

curl -N -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi slowly."}],"stream":true}' \
  https://api.openai.com/v1/chat/completions

If curl -N prints data: lines progressively and your Java program prints tokens progressively, the integration is correct. A successful run shows incremental output, terminates on [DONE], and exits with status 0.

Caveats worth knowing

BufferedReader.readLine() is fine for prototypes but allocates per line. In a high-throughput service, read chunks into a custom ByteArrayOutputStream and split on \n\n to avoid GC pressure. Also, some proxies buffer SSE; if tokens arrive in bursts, inspect the intermediary, not your Java code.

Stick to the minimal loop above and you have a robust java sse openai streaming client that compiles on any JDK 11+ with one JSON dependency.