Calling dozens of language models in parallel used to mean wrestling with fixed thread pools or chaining reactive operators. Java virtual threads llm api patterns flip that tradeoff: you write ordinary blocking methods and let the JVM schedule lightweight threads that scale to tens of thousands. This guide gives an ordered path to retrofit concurrent LLM calls into a Java 21+ service, with the pitfalls that actually bite in production.
Why virtual threads fit LLM workloads
LLM inference calls are I/O-bound and high-latency. A typical chat completion round-trip runs 200 ms to several seconds, during which the calling thread does nothing but wait on the socket.
Platform threads cost roughly 1 MB of stack each, so a pool of 200 quickly exhausts memory under bursty traffic. Java virtual threads llm api integration removes that ceiling: a virtual thread consumes only a few hundred bytes until it blocks on I/O, then yields its carrier thread.
You keep imperative control flow. No Mono.zip, no callback nesting, just loops and try/catch.
Step 1: Lock the JDK version
Virtual threads shipped as a final feature in JDK 21. Do not attempt this on JDK 17 or earlier; you will fall back to Executors.newCachedThreadPool and still pay platform-thread costs.
Set your build to release 21:
mvn clean compile -Dmaven.compiler.release=21Confirm at runtime:
java -version
# openjdk 21.0.2 2024-01-16Step 2: Write a single LLM call as a plain method
Use the standard HttpClient. Its send method blocks the caller, which is exactly what virtual threads handle well.
import java.net.http.*;
import java.net.URI;
public class LlmClient {
private final HttpClient http = HttpClient.newHttpClient();
private final String baseUrl;
private final String apiKey;
public LlmClient(String baseUrl, String apiKey) {
this.baseUrl = baseUrl;
this.apiKey = apiKey;
}
public String complete(String model, String prompt) throws Exception {
var body = """
{"model":"%s","messages":[{"role":"user","content":"%s"}]}
""".formatted(model, prompt.replace("\"", "\\\""));
var req = HttpRequest.newBuilder()
.uri(URI.create(baseUrl + "/v1/chat/completions"))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + apiKey)
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
var res = http.send(req, HttpResponse.BodyHandlers.ofString());
if (res.statusCode() != 200) {
throw new RuntimeException("LLM HTTP " + res.statusCode() + ": " + res.body());
}
return res.body();
}
}Keep the method blocking. Do not wrap it in CompletableFuture.supplyAsync unless you must interop with legacy async code.
Step 3: Launch virtual threads per task
The simplest executor is Executors.newVirtualThreadPerTaskExecutor(). It creates a new virtual thread for every submit, then tears it down on completion.
import java.util.concurrent.*;
import java.util.ArrayList;
import java.util.List;
public List<String> runBatch(LlmClient client, List<String> prompts) {
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
var futures = prompts.stream()
.map(p -> executor.submit(() -> client.complete("gpt-4o-mini", p)))
.toList();
var results = new ArrayList<String>();
for (var f : futures) {
try {
results.add(f.get());
} catch (ExecutionException e) {
results.add("ERROR: " + e.getCause().getMessage());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
return results;
}
}The try-with-resources block shuts the executor and waits for all tasks. No thread pool size to tune.
Step 4: Use structured concurrency for scoping
If you need cancellation when any single call fails, use StructuredTaskScope (standard in JDK 23, preview in JDK 21 with --enable-preview). It bounds the lifetime of child threads to a code block and propagates exceptions.
import java.util.concurrent.structured.*;
public List<String> runScoped(LlmClient client, List<String> prompts) throws Exception {
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
var tasks = prompts.stream()
.map(p -> scope.fork(() -> client.complete("gpt-4o-mini", p)))
.toList();
scope.join();
scope.throwIfFailed();
return tasks.stream().map(StructuredTaskScope.Subtask::get).toList();
}
}A failure in one fork cancels the others, which matters when you fan out to 100 models and only need a full set. Without structured concurrency, orphaned virtual threads keep running and waste sockets.
Step 5: Bound concurrency at the application layer
Virtual threads are cheap, but the LLM provider is not. Spawning 10,000 simultaneous requests will trip rate limits and return 429s.
Use a Semaphore to cap in-flight calls:
public List<String> runBounded(LlmClient client, List<String> prompts, int maxConcurrency) {
var sem = new Semaphore(maxConcurrency);
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
var futures = prompts.stream().map(p ->
executor.submit(() -> {
sem.acquire();
try { return client.complete("gpt-4o-mini", p); }
finally { sem.release(); }
})).toList();
// collect as in runBatch
} catch (InterruptedException e) { Thread.currentThread().interrupt(); }
return List.of();
}Pick maxConcurrency from the provider’s published RPM/TPM, not from heap size.
If you front requests with a gateway that provides automatic fallback when a provider is rate-limited, such as n4n.ai’s OpenAI-compatible endpoint covering 240+ models, you can raise the semaphore without writing custom retry loops. The gateway honors client routing directives and forwards provider cache-control hints, so each virtual thread either gets a response or a clean error.
Step 6: Avoid pinning the carrier thread
Virtual threads shine only when blocking calls release the carrier. Two constructs break that: synchronized blocks and native method calls that run long.
// BAD: pins carrier thread during network wait
synchronized (lock) {
return client.complete(model, prompt);
}
// GOOD: ReentrantLock is virtual-thread friendly
private final ReentrantLock lock = new ReentrantLock();
void safe() {
lock.lock();
try { /* short critical section, no I/O */ }
finally { lock.unlock(); }
}Never do CPU-heavy JSON parsing inside a lock. Offload mapping to after the response returns.
Step 7: Detect pinning in production
Enable JDK Flight Recorder and watch for jdk.VirtualThreadPinned events:
java -XX:StartFlightRecording:filename=vt.jfr -jar app.jarIf you see thousands of pinned events, a synchronized or JNI call is silently throttling throughput. Fix the hotspot before adding more load.
Step 8: Measure end-to-end, not thread count
A java virtual threads llm api service should be judged by tail latency and error rate, not by how many threads it spawned. Virtual threads make it easy to issue 5,000 parallel prompts; they do not make the model faster.
Track:
- p95 time-to-first-token if streaming
- 429 rate after your semaphore
- carrier thread pool utilization (default
ForkJoinPoolsized to CPU)
If carrier threads saturate, you are either pinning or doing CPU work inline.
Tradeoffs versus reactive stacks
Virtual threads give you readable code and standard debuggers. You lose the backpressure operators that Reactor builds in, so you must enforce limits with semaphores manually.
They are not a win for CPU-bound transforms. If you embed a local embedding model that burns 100% core, keep that on a fixed platform pool and call it from virtual threads only for the I/O wrapper.
For most Java shops calling hosted models, the java virtual threads llm api approach cuts code complexity by half with no throughput penalty.