A spring boot actuator llm health check is more than a ping endpoint when your app depends on external model APIs. If the LLM provider is degraded, your users get timeouts or 429s, and a naive readiness probe won’t catch it. This guide walks through building health indicators that actually reflect inference availability, with fallback logic and concrete tradeoffs.
1. Split liveness from readiness
Kubernetes or any orchestrator needs two signals: is the process alive, and can it serve traffic. LLM provider reachability belongs in readiness, not liveness. If you mark the pod not-ready when OpenAI is flapping, the load balancer stops sending requests, but the JVM keeps running.
Configure Actuator health groups:
management.endpoints.web.exposure.include=health
management.endpoint.health.group.readiness.include=llmCluster,diskSpace
management.endpoint.health.group.liveness.include=pingThe default ping indicator covers liveness. Your custom LLM indicator joins the readiness group only.
2. Define a provider abstraction
Don’t scatter base URLs and API keys across indicators. Wrap each backend in a small component that knows its own identity and client.
public record LLMProviderConfig(String name, String baseUrl, String apiKey, String healthModel) {}
@Configuration
class ProviderConfigs {
@Bean LLMProviderConfig openai(@Value("${llm.openai.base-url}") String u,
@Value("${llm.openai.key}") String k) {
return new LLMProviderConfig("openai", u, k, "gpt-3.5-turbo");
}
@Bean LLMProviderConfig anthropic(@Value("${llm.anthropic.base-url}") String u,
@Value("${llm.anthropic.key}") String k) {
return new LLMProviderConfig("anthropic", u, k, "claude-3-haiku-20240307");
}
}This keeps the health check code generic and testable.
3. Implement a minimal inference probe
A TCP connection to port 443 proves nothing. The provider might return 503 at the API layer. Send the cheapest valid request the API accepts. For OpenAI-compatible endpoints, a chat completion with max_tokens: 1 works:
@Component
public class ProviderHealthIndicator implements HealthIndicator {
private final WebClient client;
private final LLMProviderConfig cfg;
public ProviderHealthIndicator(WebClient.Builder b, LLMProviderConfig cfg) {
this.client = b.baseUrl(cfg.baseUrl()).build();
this.cfg = cfg;
}
@Override
public Health health() {
try {
var resp = client.post()
.uri("/v1/chat/completions")
.header("Authorization", "Bearer " + cfg.apiKey())
.bodyValue(Map.of(
"model", cfg.healthModel(),
"messages", List.of(Map.of("role","user","content","hi")),
"max_tokens", 1))
.retrieve()
.toBodilessEntity()
.block(Duration.ofSeconds(2));
if (resp != null && resp.getStatusCode().is2xxSuccessful())
return Health.up().withDetail("provider", cfg.name()).build();
return Health.down().withDetail("http", resp == null ? "null" : resp.getStatusCode().toString()).build();
} catch (Exception e) {
return Health.down().withException(e).build();
}
}
}Set the block timeout strictly. A health check that hangs for 30 seconds takes down your readiness probe slower than the provider would.
4. Cache the result to avoid self-DoS
Probing a paid LLM endpoint on every /health hit incurs token cost and latency. Actuator can be scraped every second by multiple monitors. Cache the verdict for 30 seconds:
private volatile Health cached = Health.unknown().build();
private volatile long last = 0;
private static final long TTL = 30_000;
@Override
public Health health() {
long now = System.currentTimeMillis();
if (now - last > TTL) {
cached = probe();
last = now;
}
return cached;
}If you run multiple replicas, each still probes independently—that’s fine. Don’t centralize health in a shared cache; you’d mask node-local network issues.
5. Aggregate multi-provider status
When you have more than one model backend, a single dead provider shouldn’t mark the whole service down if you can route around it. Build a composite indicator that reports DEGRADED when at least one is up:
@Component
public class LLMCluster implements HealthIndicator {
private final List<ProviderHealthIndicator> indicators;
public LLMCluster(List<ProviderHealthIndicator> indicators) {
this.indicators = indicators;
}
@Override
public Health health() {
var up = 0;
var builder = Health.status("DEGRADED");
for (var ind : indicators) {
var h = ind.health();
if (h.getStatus().equals(Status.UP)) up++;
builder.withDetail(ind.providerName(), h.getStatus().toString());
}
if (up == 0) return builder.status(Status.DOWN).build();
if (up == indicators.size()) return builder.status(Status.UP).build();
return builder.build();
}
}Register LLMCluster as the bean referenced in the readiness group, not the individual indicators.
6. Simplify with a gateway when it fits
If you front your providers with a gateway such as n4n.ai, which offers automatic fallback when a provider is rate-limited or degraded, your health check can target that single OpenAI-compatible endpoint instead of per-provider probes. You still want a DEGRADED signal if the gateway itself is unreachable, but you avoid writing multi-provider aggregation yourself. The tradeoff is an external dependency on the gateway’s routing correctness.
7. Wire security and details exposure
Health endpoints leak infrastructure topology. Lock them down:
management.endpoint.health.show-details=when_authorized
management.endpoint.health.roles=ACTUATORIf you must expose summary status publicly, use a separate /health/liveness without details. Never return provider API keys or base URLs in health details—only provider names and status codes.
8. Common pitfalls and tradeoffs
TCP-only checks. A InetAddress.isReachable or socket connect proves the network path, not the API. Providers routinely block ports while the service is 503.
Treating 429 as healthy. Rate limiting is a real degradation. If your health check sees 429, mark DEGRADED or DOWN depending on whether you have fallback. A 200 with an empty body from a misconfigured proxy is another trap—validate the response shape minimally.
Probing on the event loop. Using WebClient without block is fine, but if you block inside a reactive health indicator without a dedicated scheduler, you can stall the actuator thread. Use subscribeOn(Schedulers.boundedElastic()) or just use a @Scheduled refresh that writes to the cache.
No timeouts. Default HTTP clients wait 30–60 seconds. Always set connect and read timeouts to sub-second or low-second values for health probes.
Scraping too often. A 30-second TTL is usually enough. If your monitoring system alerts on provider outage within a minute, 30s cache gives you 30–90s detection lag—acceptable for most apps.
Ignoring regional differences. A provider might be up in us-east but down in eu-west. Run the probe from the region where the pod lives; don’t assume global status.
9. Test the indicator
Write a unit test that stubs the WebClient response and asserts DEGRADED/DOWN transitions:
@Test
void downWhen5xx() {
var cfg = new LLMProviderConfig("test", "http://localhost", "key", "m");
var client = WebClient.builder().baseUrl("http://localhost").build();
// stub with MockWebServer
var ind = new ProviderHealthIndicator(client, cfg);
// simulate 503
assertThat(ind.health().getStatus()).isEqualTo(Status.DOWN);
}Use MockWebServer from OkHttp to fake the provider. This catches header and URI mistakes before production.
10. What you get
After wiring this, GET /actuator/health/readiness returns OUT_OF_SERVICE or DOWN when all LLM paths are broken, and UP or DEGRADED when at least one works. Your deployment platform stops routing to broken pods, and your retry layer can read the same indicator to pick a live provider. That’s a spring boot actuator llm health check that earns its place in the stack.
Make sure the health model you pick is the cheapest one the provider offers. Running a 70B param inference for a ping wastes money and queue capacity. If the provider exposes a dedicated /health or /models endpoint, prefer that over a completion call—but verify it actually checks inference health, not just the gateway.
The pattern scales to any number of backends. Add a new ProviderHealthIndicator bean, drop it in the LLMCluster list, and the aggregate status updates without touching the rest of your code.