Rate limits are the default state of the world when you call hosted LLM providers from Java. This guide shows how to build a java rate limit llm api client that absorbs HTTP 429 responses, respects Retry-After, and degrades gracefully when a provider is saturated. We’ll implement a concrete OpenAI-compatible client using the standard HttpClient, then layer in backoff, circuit breaking, and gateway-level fallback.
Step 1: Set up a minimal OpenAI-compatible client
Use the built-in java.net.http.HttpClient (Java 11+). Avoid adding a heavy SDK unless you need its model types—most LLM gateways speak the OpenAI chat completions shape, which is plain JSON. If you are already on the OpenAI Java SDK, the same retry logic applies at the HttpClient layer or via a custom Interceptor.
import java.net.http.*;
import java.net.URI;
import java.time.Duration;
public class LlmClient {
private final HttpClient http = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
private final String endpoint;
private final String apiKey;
public LlmClient(String endpoint, String apiKey) {
this.endpoint = endpoint;
this.apiKey = apiKey;
}
public String chat(String model, String prompt) throws Exception {
var body = """
{"model":"%s","messages":[{"role":"user","content":"%s"}]}
""".formatted(model, prompt);
var req = HttpRequest.newBuilder()
.uri(URI.create(endpoint + "/v1/chat/completions"))
.header("Authorization", "Bearer " + apiKey)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
var resp = http.send(req, HttpResponse.BodyHandlers.ofString());
return resp.body();
}
}This is enough to succeed when limits are not hit. The java rate limit llm api problem starts the moment the provider returns 429. Define a small record for the response if you want type safety instead of raw strings:
record Usage(long prompt_tokens, long completion_tokens, long total_tokens) {}
record Choice(String text) {}
record ChatResponse(String model, Usage usage, java.util.List<Choice> choices) {}Bind with Jackson or Gson only if you need to inspect usage for metering.
Step 2: Detect rate-limit responses and parse Retry-After
A compliant LLM endpoint returns 429 Too Many Requests with a Retry-After header. The header may be a delay in seconds (Retry-After: 2) or an HTTP date (Retry-After: Wed, 21 Oct 2025 07:28:00 GMT). Parse both. Some providers also send x-ratelimit-remaining: 0 on a 200, but you should only act on 429 to avoid speculative throttling.
static Duration retryAfter(HttpResponse<?> resp) {
var ra = resp.headers().firstValue("Retry-After");
if (ra.isEmpty()) return Duration.ofSeconds(1); // default
try {
return Duration.ofSeconds(Long.parseLong(ra.get().trim()));
} catch (NumberFormatException e) {
try {
var instant = java.time.format.DateTimeFormatter
.RFC_1123_DATE_TIME.parse(ra.get(), java.time.Instant::from);
return Duration.between(java.time.Instant.now(), instant);
} catch (Exception ex) {
return Duration.ofSeconds(1);
}
}
}Do not treat 429 as a generic error. If you log it as ERROR alongside 500s, you’ll drown in noise during normal throttling. Log it at INFO or DEBUG with the model name and endpoint.
Step 3: Implement exponential backoff with jitter
Naive fixed-delay retries amplify thundering herds when many workers restart simultaneously. Use exponential backoff capped at a max, with full jitter. The Retry-After value takes precedence; we add jitter on top to avoid lockstep retries across threads.
public String chatWithRetry(String model, String prompt, int maxAttempts) throws Exception {
int attempt = 0;
while (true) {
var resp = sendRaw(model, prompt);
if (resp.statusCode() != 429) {
if (resp.statusCode() >= 500) throw new RuntimeException("Server error");
return resp.body();
}
if (++attempt >= maxAttempts) throw new RuntimeException("Rate limited after retries");
var wait = retryAfter(resp);
var base = Duration.ofMillis((long) (Math.pow(2, attempt) * 100L));
var cap = Duration.ofSeconds(30);
var backoff = base.compareTo(cap) > 0 ? cap : base;
var jitter = Duration.ofMillis((long) (Math.random() * backoff.toMillis()));
var sleep = wait.plus(jitter);
Thread.sleep(sleep.toMillis());
}
}
private HttpResponse<String> sendRaw(String model, String prompt) throws Exception {
var body = """
{"model":"%s","messages":[{"role":"user","content":"%s"}]}
""".formatted(model, prompt);
var req = HttpRequest.newBuilder()
.uri(URI.create(endpoint + "/v1/chat/completions"))
.header("Authorization", "Bearer " + apiKey)
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
return http.send(req, HttpResponse.BodyHandlers.ofString());
}On Java 21+, prefer virtual threads for the caller so a sleeping retry does not occupy a platform thread:
try (var executor = java.util.concurrent.Executors.newVirtualThreadPerTaskExecutor()) {
var future = executor.submit(() -> chatWithRetry("gpt-4o-mini", "hi", 5));
return future.get();
}Step 4: Add a circuit breaker to shed load
If a provider is consistently throttling, retrying locally wastes threads and tokens. Use Resilience4j or a simple in-memory counter. Below is a minimal manual breaker that opens after N consecutive failures and cools down for a fixed period.
import java.util.concurrent.atomic.*;
public class Breaker {
private final AtomicInteger failures = new AtomicInteger(0);
private volatile long openUntil = 0;
private final int threshold;
private final Duration cooldown;
public Breaker(int threshold, Duration cooldown) {
this.threshold = threshold; this.cooldown = cooldown;
}
public boolean allow() {
return System.currentTimeMillis() > openUntil;
}
public void onSuccess() { failures.set(0); }
public void onFailure() {
if (failures.incrementAndGet() >= threshold) {
openUntil = System.currentTimeMillis() + cooldown.toMillis();
failures.set(0);
}
}
}Wrap the call:
if (!breaker.allow()) throw new RuntimeException("Circuit open");
try {
var out = chatWithRetry(model, prompt, 3);
breaker.onSuccess();
return out;
} catch (Exception e) {
breaker.onFailure();
throw e;
}If you prefer a library, Resilience4j’s CircuitBreaker decorator composes cleanly with HttpClient via a ClientHttpResponse interceptor.
Step 5: Offload fallback to an inference gateway
Writing multi-provider failover in Java is tedious: you must map model names, handle differing error shapes, and keep credentials for each. A gateway like n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and performs automatic fallback when a provider is rate-limited or degraded. Point your LlmClient at that single endpoint and the java rate limit llm api logic on the client shrinks to a thin retry wrapper—the gateway redistributes the request server-side to a healthy provider.
var client = new LlmClient("https://api.n4n.ai", System.getenv("N4N_KEY"));The gateway returns the same usage block, so your token accounting does not change. You still keep the local circuit breaker to protect against gateway-wide outages, but you can drop any custom provider-priority list.
Step 6: Forward cache-control and routing directives
If you call providers directly, honor their cache hints. OpenAI-compatible APIs accept cache_control in the body for prompt caching. When using a gateway, it forwards provider cache-control hints, so you set them once and they propagate.
{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "You are a helpful assistant", "cache_control": {"type": "ephemeral"}}
]
}Ignoring these wastes money and increases token rate limit pressure because the provider recomputes the prompt prefix on every call.
Step 7: Verify with a stubbed rate-limit server
Do not test against the live API—you’ll get blocked. Use WireMock to return 429 with Retry-After: 0 then 200. This proves your retry loop actually waits and recovers.
// WireMock stub
stubFor(post(urlEqualTo("/v1/chat/completions"))
.inScenario("ratelimit")
.whenScenarioStateIs(STARTED)
.willReturn(aResponse().withStatus(429).withHeader("Retry-After","0"))
.willSetStateTo("ok"));
stubFor(post(urlEqualTo("/v1/chat/completions"))
.inScenario("ratelimit")
.whenScenarioStateIs("ok")
.willReturn(aResponse().withStatus(200)
.withBody("{\"choices\":[{\"message\":{\"content\":\"hi\"}}]}")));A JUnit test:
@Test
void retriesOn429() throws Exception {
var client = new LlmClient("http://localhost:8080", "test");
var out = client.chatWithRetry("gpt-4o-mini", "ping", 3);
assertTrue(out.contains("hi"));
}Run the test, assert it passes in under a second (because Retry-After: 0), and confirm the server logged exactly one 429. That validates the java rate limit llm api retry path without touching production.
Step 8: Production checklist
- Emit metrics on 429 count per model and per endpoint; alert on sustained circuit-open events.
- Set a hard deadline on total retry time (e.g., 2 minutes max) to bound tail latency.
- Use a shared
HttpClientwith a bounded connection pool; do not create one per request. - If you use n4n.ai, per-token usage metering is handled at the gateway, so read
usage.total_tokensfrom the response instead of counting locally. - Never retry on 4xx other than 429 (e.g., 401 means bad key, 400 means malformed request).
- Keep
maxAttemptslow (3–5); beyond that, fail fast and let the caller degrade.
A java rate limit llm api client that implements these steps will survive provider throttling without taking down your service. The pattern is boring on purpose: detect, back off, break, and let a gateway handle the rest.