Most LLM SDKs for Java block the calling thread on HTTP I/O. A well-designed java completablefuture llm client lets you pipeline prompts, fan out to multiple models, and absorb provider latency without dedicating a thread per in-flight request. This guide builds a production-shaped client from the standard library and nothing else.
Step 1: Pick a single OpenAI-compatible endpoint
You do not want to hardcode provider URLs and auth in your async chains. Point your client at one OpenAI-compatible base URL and pass the model name in the body. A gateway that fronts many providers saves you from writing fallback logic; for example, n4n.ai exposes one endpoint covering 240+ models and automatically reroutes when a provider is rate-limited, so your CompletableFuture pipeline stays clean.
Set two constants and treat them as configuration, not literals:
String BASE_URL = "https://api.n4n.ai/v1/chat/completions"; // or your own proxy
String API_KEY = System.getenv("LLM_API_KEY");If you run your own proxy, swap the URL. The rest of the code is identical. The key win is that your java completablefuture llm client never branches on which vendor is behind the model string.
Step 2: Use java.net.http.HttpClient in async mode
The built-in HttpClient (Java 11+) returns CompletableFuture<HttpResponse<String>> from sendAsync. Do not wrap a synchronous send inside CompletableFuture.supplyAsync—that just moves blocking onto a worker thread and defeats the purpose. Use the native async method and keep one client instance for the JVM.
HttpClient http = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(5))
.followRedirects(HttpClient.Redirect.NORMAL)
.build();The client pools connections and dispatches completion callbacks on its own internal executor. For very high fan-out, pass a bounded Executor via .executor(Executors.newFixedThreadPool(8)) so you control the callback thread count instead of relying on the default.
Step 3: Define request and response records
Model the wire format with records. Avoid pulling in a heavy SDK just for POJOs.
record Message(String role, String content) {}
record ChatRequest(String model, List<Message> messages, double temperature) {}
record Choice(Message message) {}
record ChatResponse(List<Choice> choices) {}A typical JSON body looks like this:
{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello"}],
"temperature": 0.7
}Serialize with Jackson or Gson. Keep the ObjectMapper as a static singleton; it is thread-safe after configuration.
ObjectMapper mapper = new ObjectMapper();
String body = mapper.writeValueAsString(
new ChatRequest("gpt-4o-mini", List.of(new Message("user", "Hello")), 0.7));Step 4: Build the core completion method
Write a method that returns CompletableFuture<ChatResponse>. It constructs the request, sends async, and maps the body. This is the heart of your java completablefuture llm client.
public CompletableFuture<ChatResponse> complete(ChatRequest req) {
String json;
try { json = mapper.writeValueAsString(req); }
catch (JsonProcessingException e) { return CompletableFuture.failedFuture(e); }
HttpRequest httpReq = HttpRequest.newBuilder()
.uri(URI.create(BASE_URL))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + API_KEY)
.POST(HttpRequest.BodyPublishers.ofString(json))
.build();
return http.sendAsync(httpReq, HttpResponse.BodyHandlers.ofString())
.thenApply(res -> {
if (res.statusCode() != 200) {
throw new RuntimeException("HTTP " + res.statusCode() + ": " + res.body());
}
try { return mapper.readValue(res.body(), ChatResponse.class); }
catch (JsonProcessingException e) { throw new RuntimeException(e); }
});
}The thenApply stage runs on the HTTP client’s callback thread. Do not do heavy CPU work there; if you need to parse large responses or embed vectors, shift to a custom executor with thenApplyAsync.
Step 5: Compose sequential pipelines
Chain a follow-up call without blocking by using thenCompose. Suppose you want a haiku, then a translation:
CompletableFuture<String> translated = complete(
new ChatRequest("gpt-4o-mini",
List.of(new Message("user", "Write a haiku about latency")), 0.8))
.thenCompose(first -> {
String text = first.choices().get(0).message().content();
ChatRequest second = new ChatRequest("gpt-4o-mini",
List.of(new Message("user", "Translate to French: " + text)), 0.3);
return complete(second);
})
.thenApply(r -> r.choices().get(0).message().content());The two LLM calls run back-to-back on the async callback thread. No join() in the middle. Extract the inner lambda to a method if the chain grows:
private CompletableFuture<ChatResponse> translateStep(ChatResponse first) { ... }This keeps the pipeline readable and testable.
Step 6: Fan out to multiple models in parallel
Use CompletableFuture.allOf to query several models and collect outputs. This is where the async design pays off—three network calls become one wait window.
List<String> models = List.of("gpt-4o", "claude-3-5-sonnet", "mistral-large");
List<CompletableFuture<ChatResponse>> calls = models.stream()
.map(m -> complete(new ChatRequest(m,
List.of(new Message("user", "Ping")), 0.0))
.exceptionally(ex -> new ChatResponse(List.of(
new Choice(new Message("assistant", "ERROR"))))))
.toList();
CompletableFuture<Void> all = CompletableFuture.allOf(calls.toArray(new CompletableFuture[0]));
CompletableFuture<List<String>> results = all.thenApply(v ->
calls.stream()
.map(f -> f.join().choices().get(0).message().content())
.toList());Isolating failures per call with exceptionally prevents one bad provider from failing the whole batch. The join() inside thenApply is safe because allOf guarantees all futures are done.
Step 7: Add timeouts and fallbacks
Providers stall. Use orTimeout (Java 9+) to bound wait time, then recover with exceptionally.
CompletableFuture<ChatResponse> safeCall = complete(req)
.orTimeout(10, TimeUnit.SECONDS)
.exceptionally(ex -> new ChatResponse(List.of(
new Choice(new Message("assistant", "{\"error\":\"timeout\"}")))));If you need retries, write a recursive helper:
public CompletableFuture<ChatResponse> completeWithRetry(ChatRequest req, int tries) {
return complete(req).exceptionally(ex -> null)
.thenCompose(r -> {
if (r != null) return CompletableFuture.completedFuture(r);
if (tries <= 1) return CompletableFuture.failedFuture(
new RuntimeException("exhausted retries"));
return completeWithRetry(req, tries - 1);
});
}A java completablefuture llm client should push retry and timeout to the edge of the pipeline, not scatter Thread.sleep through business logic.
Step 8: Run and verify success
Write a main that exercises the pipeline and prints. Success means you see non-null content and the JVM exits without hanging.
public static void main(String[] args) {
LlmClient client = new LlmClient();
client.complete(new ChatRequest("gpt-4o-mini",
List.of(new Message("user", "Say hi in JSON")), 0.1))
.thenAccept(r -> System.out.println(r.choices().get(0).message().content()))
.join(); // only block at the process edge
}Verify by running java LlmClient.java (or mvn exec:java). Check that the printed string is valid JSON. If you wired the fan-out step, confirm three outputs appear and total runtime is under the slowest single call plus overhead—not the sum of all three. For automated verification, drop the join() into a JUnit test with a 15-second timeout:
assertTimeout(Duration.ofSeconds(15), () ->
client.complete(testReq).join().choices().isEmpty());Practical notes
- Reuse the
HttpClientandObjectMapper; both are expensive to build. - Map transport errors to domain exceptions before
thenApplyto keep pipelines clean. - If you need cancellation, pass a
CancellationTokentosendAsyncand propagate it from your service layer. - For high throughput, set a custom
Executoron the client and usethenApplyAsyncfor CPU-heavy post-processing.
A java completablefuture llm client built this way scales to thousands of concurrent prompts on a few carrier threads. It is boring, standard, and ships.