Building a spring boot reactive retry llm integration that survives flaky provider networks requires more than a try-catch block. You need backpressure-aware retries, exponential backoff with jitter, and a fallback path for when the model endpoint returns 429 or 503. This guide walks through a complete Reactor-based implementation you can drop into a Spring Boot 3 service and test end to end.
Step 1: Configure a reactive WebClient for LLM requests
Start by pulling the WebFlux starter. Do not mix RestTemplate into a reactive controller; it allocates a thread per call and silently kills throughput under load.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>io.projectreactor.addons</groupId>
<artifactId>reactor-extra</artifactId>
</dependency>Define a WebClient bean with explicit timeouts. LLM endpoints can stream for 30–60 seconds; a default 30s response timeout will cut off valid completions. Use a short connect timeout and a long read timeout.
@Configuration
public class LlmClientConfig {
@Bean
public WebClient llmWebClient(WebClient.Builder builder) {
HttpClient httpClient = HttpClient.create()
.connectTimeout(Duration.ofSeconds(2))
.responseTimeout(Duration.ofSeconds(60));
return builder
.baseUrl("https://api.n4n.ai/v1") // OpenRouter-class gateway, one endpoint for 240+ models
.clientConnector(new ReactorClientHttpConnector(httpClient))
.build();
}
}Keep the base URL in application.yml so you can switch environments without recompiling:
llm:
base-url: https://api.n4n.ai/v1
default-model: gpt-4o-miniThe spring boot reactive retry llm pattern assumes every outbound call flows through this single WebClient instance. Reusing the client preserves connection pools and makes retry instrumentation uniform.
Step 2: Implement the core retry operator
Project Reactor’s retryWhen is the only correct retry primitive for reactive pipelines. The naive .retry(5) resubscribes instantly and will spike a degraded provider into a full outage. Use Retry.backoff from reactor-extra, which applies exponential delay with optional jitter.
Below is a complete service class skeleton. It posts to an OpenAI-compatible /chat/completions route and retries only on transient conditions.
@Service
public class LlmService {
private final WebClient llmWebClient;
private final MeterRegistry metrics;
public LlmService(WebClient llmWebClient, MeterRegistry metrics) {
this.llmWebClient = llmWebClient;
this.metrics = metrics;
}
public Mono<String> completePrompt(String prompt, String model) {
return llmWebClient.post()
.uri("/chat/completions")
.bodyValue(Map.of(
"model", model,
"messages", List.of(Map.of("role", "user", "content", prompt))))
.retrieve()
.bodyToMono(String.class)
.retryWhen(buildRetryPolicy())
.onErrorResume(e -> fallbackCompletion(prompt, e));
}
private Retry<?> buildRetryPolicy() {
return Retry.backoff(5, Duration.ofMillis(200))
.maxBackoff(Duration.ofSeconds(2))
.jitter(0.5)
.filter(this::isRetryable)
.doOnRetry(s -> metrics.counter("llm.retry",
"attempt", String.valueOf(s.totalRetries())).increment());
}
private boolean isRetryable(Throwable t) {
if (t instanceof WebClientResponseException ex) {
int status = ex.getStatusCode().value();
return status == 429 || status == 503 || status == 500;
}
return t instanceof IOException
|| t instanceof TimeoutException
|| t instanceof PrematureCloseException;
}
}The filter rejects 4xx client errors (except 429) because retrying a malformed request is wasted work. The base delay of 200ms with a cap of 2s and 50% jitter means the fifth attempt waits roughly 1.6s–2.4s. That keeps tail latency bounded while absorbing short provider blips.
A spring boot reactive retry llm flow that omits the maxBackoff cap can accidentally wait minutes under repeated failures. Always cap.
Step 3: Add a fallback to protect callers
After five retries the pipeline hits onErrorResume. Returning a raw exception to a user-facing endpoint is poor UX. Provide a fallback that degrades gracefully: a smaller model, a cached answer, or a static string.
private Mono<String> fallbackCompletion(String prompt, Throwable e) {
log.warn("LLM call failed after retries: {}", e.toString());
return llmWebClient.post()
.uri("/chat/completions")
.bodyValue(Map.of(
"model", "mistral-7b-instruct",
"messages", List.of(Map.of("role", "user", "content", prompt))))
.retrieve()
.bodyToMono(String.class)
.timeout(Duration.ofSeconds(10))
.onErrorReturn("{\"error\":\"model unavailable\"}");
}If you front your calls with a gateway like n4n.ai, the server side already shifts to a healthy provider when a model is rate-limited or degraded. Client-side retry still matters for connection resets, DNS hiccups, and local pod network issues. The two layers are complementary, not redundant.
For batch jobs, the fallback might write the prompt to a dead-letter queue instead of calling another model. Design the fallback signature so it can be swapped without touching the retry logic.
Step 4: Write a test that proves retries fire
Use MockWebServer from OkHttp to simulate a flaky upstream. The test below enqueues two 503 responses and then a 200, then asserts the client recovered after exactly three requests.
@SpringBootTest
class LlmServiceTest {
private MockWebServer mockWebServer;
private LlmService llmService;
@BeforeEach
void setUp() throws IOException {
mockWebServer = new MockWebServer();
mockWebServer.start();
WebClient client = WebClient.builder()
.baseUrl(mockWebServer.url("/").toString())
.build();
llmService = new LlmService(client, new SimpleMeterRegistry());
}
@AfterEach
void tearDown() throws IOException {
mockWebServer.shutdown();
}
@Test
void retriesOn503ThenSucceeds() {
mockWebServer.enqueue(new MockResponse().setResponseCode(503));
mockWebServer.enqueue(new MockResponse().setResponseCode(503));
mockWebServer.enqueue(new MockResponse()
.setBody("{\"ok\":true}").setResponseCode(200));
String result = llmService.completePrompt("hi", "test-model").block();
assertThat(result).contains("ok");
assertThat(mockWebServer.getRequestCount()).isEqualTo(3);
}
}Run with ./mvnw test. A green test proves your spring boot reactive retry llm pipeline retries exactly as configured and does not loop infinitely. Add a second test that enqueues six 503s and asserts the fallback response is returned.
Timing verification
To confirm backoff, record request timestamps in the test:
List<Long> times = new ArrayList<>();
mockWebServer.setDispatcher(new Dispatcher() {
@Override public MockResponse dispatch(RecordedRequest r) {
times.add(System.nanoTime());
return new MockResponse().setResponseCode(503);
}
});After the call resolves, assert that times.get(1) - times.get(0) is at least 200ms. This catches accidental zero-delay retries.
Step 5: Instrument with Micrometer and tracing
Retries hide failures from callers but must be visible to operators. The doOnRetry hook in Step 2 already increments a counter. Add a timer around the whole call:
public Mono<String> completePrompt(String prompt, String model) {
return Mono.timer(Duration.ZERO)
.then(llmWebClient.post() /* ... same as before ... */)
.name("llm.completion")
.metrics();
}With Reactor’s metrics() operator, you get subscribe/terminate timers in Prometheus. Pair this with a Reactor Context write of the traceId so retries appear under the same span in Zipkin.
.contextWrite(ctx -> ctx.put("traceId", TraceContext.current().traceId()))A rising llm_retry count with stable llm_completion latency means the provider is intermittently throttling. A rising retry count with soaring latency means the fallback is also struggling—time to alert.
Step 6: Honor provider cache-control and routing directives
OpenAI-compatible gateways forward cache hints. If your prompt is static (e.g., a system instruction), set Cache-Control to let the gateway reuse a provider-side cached completion. This cuts 429s and token cost.
llmWebClient.post()
.uri("/chat/completions")
.headers(h -> h.setCacheControl(CacheControl.maxAge(1, TimeUnit.HOURS)))
.bodyValue(/* ... */)When you send x-routing: prefer=anthropic, the gateway honors it. Your retry filter must not treat a 400 from an invalid routing header as retryable—that is a bug in your code, not a transient fault. Keep the isRetryable method strict.
Step 7: Production checklist
Before shipping, verify:
- All model calls go through the retrying
completePromptmethod; no strayWebClientcalls bypass it. - Fallback model is cheaper and faster; otherwise you amplify load during incidents.
maxBackoffis set; default unbounded backoff can wedge a pod.- Metrics dashboard shows retry rate per model.
- Integration test runs in CI with
MockWebServerto block regressions.
Verify success
Run the service locally and send a request through a controller:
curl -X POST localhost:8080/complete \
-d '{"prompt":"explain retry"}' \
-H 'Content-Type: application/json'Point the llm.base-url at a blackhole (e.g., http://127.0.0.1:9) and re-run. Logs should show backoff warnings, then the fallback JSON. Restore the URL; the earlier MockWebServer test already proved the happy path. If both behave, your spring boot reactive retry llm implementation is production-ready.
Common pitfalls
- Blocking inside
map: never call.block()in a reactive chain; useflatMapand returnMono. - Retrying on 400: client errors waste attempts and mask bugs.
- Ignoring timeouts: a hung connection without
responseTimeoutstalls the pipeline indefinitely. - No jitter: synchronized retries across pods cause periodic provider spikes.
- Swallowing errors in fallback: log the original exception; otherwise incidents go dark.
Build the retry policy once as a shared Retry bean, reuse it across every model call, and your Spring Boot service will stay responsive when the LLM provider does not.