Most Java services that call an LLM endpoint open a new HTTP connection per request and wonder why p99 latency climbs under load. Proper java connection pooling llm api clients reuse TCP connections and keep-alive sockets, turning a flood of requests into a bounded, manageable stream.
Pick a client that actually pools
Java’s built-in java.net.http.HttpClient (available since Java 11) maintains a process-wide connection pool, but exposes almost no knobs for sizing, per-route limits, or idle eviction. For high-throughput LLM integration you want explicit control. Use Apache HttpClient 5 or OkHttp 4. Both provide configurable pools, idle connection reaping, and separate dispatch threads.
OkHttp’s ConnectionPool is minimal and predictable:
import okhttp3.*;
import java.util.concurrent.*;
import java.util.concurrent.TimeUnit;
ConnectionPool pool = new ConnectionPool(200, 5, TimeUnit.MINUTES);
OkHttpClient client = new OkHttpClient.Builder()
.connectionPool(pool)
.dispatcher(new Dispatcher(new ThreadPoolExecutor(
0, 400, 60, TimeUnit.SECONDS,
new SynchronousQueue<>())))
.build();Apache HttpClient 5 gives per-route limits, which matter if you call multiple model hosts through one client:
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManagerBuilder;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.classic.CloseableHttpClient;
var cm = PoolingHttpClientConnectionManagerBuilder.create()
.setMaxConnTotal(500)
.setMaxConnPerRoute(200)
.setConnectionTimeToLive(5, TimeUnit.MINUTES)
.build();
CloseableHttpClient client = HttpClients.custom()
.setConnectionManager(cm)
.build();Size the pool to your concurrency, not your cores
Engineers often size LLM connection pools like Postgres pools—a few per CPU. That fails because LLM calls are I/O bound and slow. A chat completion might take 800 ms; a long generation can run 20–30 s.
Calculate required connections from target throughput and latency:
maxConnections = targetRPS * avgLatencySeconds * 1.2If you need 150 RPS at a 2 s mean latency, you need roughly 360 concurrent connections. Set maxConnPerRoute (or OkHttp’s idle cap plus dispatcher max) to that value. Split across providers if you shard.
Tradeoff: every idle socket holds a file descriptor and a kernel TCP buffer. Linux default ulimit -n is frequently 1024. Raise it before you scale, or the JVM will throw SocketException: too many open files.
Separate timeouts from pool limits
A pooled connection blocked on a slow read occupies a slot. If your read timeout is 60 s but the model usually returns in 3 s, a handful of stalled streams exhaust the pool and queue everything else. Set connect, read, and total call timeouts independently.
OkHttpClient client = new OkHttpClient.Builder()
.connectTimeout(2, TimeUnit.SECONDS)
.readTimeout(30, TimeUnit.SECONDS)
.callTimeout(35, TimeUnit.SECONDS)
.build();For Apache, set ConnectionRequestTimeout (how long to wait for a free connection) and ResponseTimeout on the request config. Fail fast when the pool is saturated; let the caller shed load rather than block indefinitely.
Reuse the client and close bodies
Never instantiate a new OkHttpClient or CloseableHttpClient per request. The pool lives inside the client instance. Equally critical: always close the response body, even on non-200 responses, or the socket is not returned to the pool.
try (Response resp = client.newCall(req).execute()) {
if (!resp.isSuccessful()) {
throw new RuntimeException("LLM call failed: " + resp.code());
}
String output = resp.body().string();
}With streaming endpoints, drain the full stream before closing, or use resp.body().source().readAll(sink) to ensure the connection is cleanly released.
Let the gateway handle fallback
If you front your models with an OpenAI-compatible gateway such as n4n.ai, which performs automatic fallback when a provider is rate-limited or degraded, you can point a single pooled client at one endpoint and let the gateway handle routing. This collapses your java connection pooling llm api logic to a single host pool and removes client-side retry storms. You still set timeouts, but you avoid per-provider pools and custom circuit breakers.
Streaming changes pool math
Many LLM calls use Server-Sent Events. The TCP connection stays open until the final token, so pool occupancy equals time-to-first-token plus full generation time. Fifty concurrent 20 s generations require 50 free connections minimum—on top of any non-streaming traffic.
Consider a dedicated ConnectionPool (or separate HttpClient) for streaming versus fast embedding or classification calls. That prevents a slow generative stream from starving a 200 ms embedding request.
Kill stale connections
Load balancers and NATs drop idle TCP sessions silently. OkHttp only validates a connection on checkout; Apache can validate after inactivity. Configure both to evict idle sockets aggressively and, for Apache, validate periodically:
cm.setValidateAfterInactivity(5_000); // millisecondsFor OkHttp, run a background task that calls pool.evictAll() on a health endpoint, or rely on the 5-minute keep-alive with short TTL.
Watch DNS caching
The JVM caches DNS resolutions forever by default (networkaddress.cache.ttl = -1). If your LLM endpoint is behind a gateway that rotates IPs, you will pin to a dead node. Set a finite TTL:
System.setProperty("networkaddress.cache.ttl", "30");Or install a custom Dns resolver in OkHttp that respects upstream TTLs.
Monitor what the pool tells you
OkHttp does not expose pool stats directly, but you can subclass EventListener to record connection acquired/released events, or poll ConnectionPool.idleConnectionCount(). Apache exposes cm.getTotalStats() with getLeased(), getAvailable(), and getPending(). Export to Prometheus:
llm_pool_leased_connectionsllm_pool_idle_connectionsllm_dispatcher_queued_calls
When queued calls climb, either the pool is too small or upstream latency spiked. Alert on pending > 0 for sustained periods.
Minimal working example
A compact OkHttp setup for a service calling one LLM endpoint at ~150 RPS with 1.5 s mean latency:
public final class LlmClient {
private static final OkHttpClient CLIENT = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(300, 5, TimeUnit.MINUTES))
.dispatcher(new Dispatcher(new ThreadPoolExecutor(
0, 600, 60, TimeUnit.SECONDS,
new SynchronousQueue<>())))
.connectTimeout(2, TimeUnit.SECONDS)
.readTimeout(30, TimeUnit.SECONDS)
.callTimeout(32, TimeUnit.SECONDS)
.addInterceptor(chain -> {
Request r = chain.request().newBuilder()
.header("Authorization", "Bearer " + System.getenv("LLM_KEY"))
.build();
return chain.proceed(r);
})
.build();
public static String complete(String prompt) throws IOException {
MediaType json = MediaType.get("application/json");
String body = "{\"model\":\"gpt-4o\",\"messages\":[{\"role\":\"user\",\"content\":\"" + prompt + "\"}]}";
Request req = new Request.Builder()
.url("https://api.example.com/v1/chat/completions")
.post(RequestBody.create(json, body))
.build();
try (Response resp = CLIENT.newCall(req).execute()) {
if (!resp.isSuccessful()) throw new IOException("status " + resp.code());
return resp.body().string();
}
}
}This keeps 300 idle sockets alive, allows up to 600 concurrent dispatches, and bounds every call at 32 s.
Tradeoffs and final notes
Connection pooling cuts TLS handshake and TCP slow-start overhead, but it does not reduce model latency. If a provider throttles you, a larger pool just creates a longer wait queue. Apply backpressure: cap the dispatcher executor and let callers fail or shed load instead of blocking.
For Java services, the fastest path to stable high-throughput LLM integration is one well-tuned pool, explicit timeouts, and a gateway that absorbs provider volatility. That keeps your java connection pooling llm api code boring—which is exactly what you want when traffic spikes at 3 a.m.