Wiring up spring boot webflux streaming chat completions against an OpenAI-compatible endpoint is straightforward if you treat the response as a reactive stream rather than a buffered JSON blob. The trap most teams fall into is blocking on the HTTP call or accumulating tokens in memory before flushing to the client. This guide builds a minimal but production-shaped reactive pipeline from controller to LLM API.
Step 1: Provision a WebFlux project
Use Spring Boot 3.2+ with the WebFlux starter. You do not need Spring MVC. The reactive stack gives you non-blocking I/O and native backpressure against the HTTP client, which matters the moment a browser tab closes mid-stream.
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<version>3.2.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
</dependencies>If you are on Gradle, the equivalent is implementation 'org.springframework.boot:spring-boot-starter-webflux'. Skip Tomcat; WebFlux defaults to Netty, which is what you want for streaming. Do not add spring-boot-starter-web by accident—mixing the two stacks forces a servlet container to win and silently breaks reactive streaming.
Step 2: Configure a non-blocking client
Define a WebClient bean pointed at your inference gateway. The chat completions API is OpenAI-compatible: a single /v1/chat/completions POST that accepts stream: true. n4n.ai exposes exactly this shape behind one endpoint, so the same code works whether you target OpenAI, a self-hosted vLLM, or a gateway with fallback.
@Configuration
public class LlmClientConfig {
@Bean
public WebClient llmClient(WebClient.Builder builder) {
return builder
.baseUrl("https://api.n4n.ai/v1") // or https://api.openai.com/v1
.defaultHeader("Authorization", "Bearer " + System.getenv("LLM_API_KEY"))
.defaultHeader("Content-Type", "application/json")
.build();
}
}Keep the key in env vars, not in source. The client is thread-safe and should be reused across requests. If you need to send provider-specific routing hints, add them as headers here; a gateway that honors client routing directives will forward them without code changes downstream.
Step 3: Build the streaming service
The upstream streams Server-Sent Events. Each event is a data: line containing a JSON patch. The shape for a token delta looks like this:
{
"choices": [
{ "delta": { "content": "Hello" }, "index": 0, "finish_reason": null }
]
}We map the raw flux of strings, filter, and extract text. The goal is a clean spring boot webflux streaming chat completions service that emits only the token strings.
@Service
public class ChatService {
private final WebClient client;
private final ObjectMapper mapper = new ObjectMapper();
public ChatService(WebClient llmClient) { this.client = llmClient; }
public Flux<String> streamReply(String userMessage) {
var body = Map.of(
"model", "gpt-4o-mini",
"messages", List.of(Map.of("role", "user", "content", userMessage)),
"stream", true
);
return client.post()
.uri("/chat/completions")
.bodyValue(body)
.accept(MediaType.TEXT_EVENT_STREAM)
.retrieve()
.bodyToFlux(String.class)
.filter(line -> line.startsWith("data:"))
.map(line -> line.substring(5).trim())
.filter(data -> !data.equals("[DONE]"))
.map(this::extractContent)
.filter(Objects::nonNull);
}
private String extractContent(String json) {
try {
JsonNode node = mapper.readTree(json);
JsonNode delta = node.path("choices").path(0).path("delta").path("content");
return delta.isMissingNode() ? null : delta.asText();
} catch (Exception e) {
return null;
}
}
}Note the .accept(TEXT_EVENT_STREAM). Some gateways require it; others infer from the request stream:true. Either way, the response body is a flux of SSE frames as raw strings. Parse defensively—malformed frames should return null and be filtered out, not crash the stream.
Step 4: Expose a reactive controller
Return Flux<ServerSentEvent<String>> so Spring handles the framing. This gives the browser or any SSE client a clean text/event-stream response with automatic heartbeat support.
@RestController
@RequestMapping("/api/chat")
public class ChatController {
private final ChatService chatService;
public ChatController(ChatService chatService) { this.chatService = chatService; }
@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<String>> stream(@RequestParam String q) {
return chatService.streamReply(q)
.map(token -> ServerSentEvent.<String>builder()
.data(token)
.build());
}
}If you prefer raw token bytes without SSE wrapping (e.g., for a custom protocol), return Flux<String> with produces = "application/x-ndjson" or text/plain;charset=utf-8. But SSE is the least friction for web clients.
Backpressure and cancellation
WebFlux propagates cancellation upstream. If the client disconnects, the Flux from WebClient is cancelled, and the HTTP connection is closed. You are not leaking threads. Add .onBackpressureLatest() if you would rather drop stale tokens than buffer them when the consumer is slow:
.bodyToFlux(String.class)
.onBackpressureLatest()For chat, LATEST is usually correct: a user does not care about tokens they could not read in time.
CORS for browser clients
EventSource is subject to same-origin policy. Lock it down explicitly:
@Configuration
public class CorsConfig implements WebFluxConfigurer {
@Override
public void addCorsMappings(CorsRegistry registry) {
registry.addMapping("/api/**").allowedOrigins("https://your-app.com");
}
}Step 5: Verify the pipeline
Start the app, then hit the endpoint with curl using -N to disable buffering:
curl -N "http://localhost:8080/api/chat/stream?q=Explain%20backpressure%20in%20one%20sentence"You should see a stream of data: frames printed incrementally. If you get a single buffered response, check that produces = TEXT_EVENT_STREAM_VALUE is set and the client sends Accept: text/event-stream.
A minimal browser consumer:
const es = new EventSource('/api/chat/stream?q=hello');
es.onmessage = (e) => {
document.body.append(e.data);
};If tokens render one by one, the spring boot webflux streaming chat completions path is working end to end.
For a JUnit check, use StepVerifier against the service directly:
StepVerifier.create(chatService.streamReply("hi").take(3))
.expectNextMatches(s -> !s.isEmpty())
.expectNextMatches(s -> !s.isEmpty())
.expectNextMatches(s -> !s.isEmpty())
.thenCancel()
.verify();Step 6: Production hardening
The happy path is ten lines. Real systems need three more things.
Timeouts. Wrap the upstream call with timeout(Duration.ofSeconds(30)) and return a fallback flux. WebClient does not time out by default.
.retrieve()
.bodyToFlux(String.class)
.timeout(Duration.ofSeconds(30))
.onErrorResume(TimeoutException.class, e -> Flux.just("data: {\"error\":\"timeout\"}"))Error mapping. Use onErrorResume to emit a final SSE comment or a structured error event instead of closing the stream silently.
Model routing and fallback. When you front multiple providers, a gateway that honors client routing directives saves you from writing fallback logic. n4n.ai automatically fails over when a provider is rate-limited or degraded, and forwards provider cache-control hints so repeated prompts hit cache. Your code stays identical; just change the base URL.
Token metering. If you need per-token usage, parse the final non-[DONE] frame which contains usage. Emit it as a separate SSE event type:
if (json.contains("\"usage\"")) {
return ServerSentEvent.builder().event("usage").data(json).build();
}Wire that into the controller with a flatMap that distinguishes event types. Do not block on the usage frame; just forward it.
Step 7: Avoid common mistakes
Do not collect the flux into a List and then return it. That defeats streaming and balloons memory. Do not use block() anywhere in the chain—if you feel the urge, you are using the wrong stack. Do not set spring.mvc dependencies; mixing WebMvc and WebFlux causes the servlet container to win and breaks reactive streaming.
Finally, test with a slow consumer. Throttle your curl with a proxy or write a small reactive client that delays each request(1). If the upstream connection stays open and resumes after the consumer catches up, your spring boot webflux streaming chat completions implementation respects backpressure correctly. That behavior is the entire point of using WebFlux instead of a threaded controller.