Connection pooling for LLM APIs with reqwest

When you call an LLM API from Rust, every request travels over HTTPS. Without rust reqwest connection pooling, each call opens a fresh TCP socket and performs a TLS handshake, adding tens to hundreds of milliseconds of latency and wasting file descriptors. This guide shows how to build a pooled client once, tune it for concurrent streaming workloads, and avoid the mistakes that silently defeat reuse.

Why pooling changes your tail latency

LLM inference requests are slow by nature—often 200 ms to many seconds for generation. The connection setup overhead is small relative to generation but becomes significant when you fire thousands of requests per minute or use short prompts with streaming. A pooled client keeps established connections alive and reuses them for subsequent requests to the same host.

Under the hood, reqwest uses hyper’s connection pool. Hyper holds idle connections in a per-host structure, but the default pool_max_idle_per_host is 1 in older hyper versions and still conservative in current ones. That means after a request finishes, only one connection stays open; the next concurrent request must open another. Under load you still spin up sockets, pay TLS again, and incur TCP slow-start penalties on every new connection.

For a service that issues 100 requests per second to a single model endpoint, a missing pool config can mean 100 TLS handshakes per second. That is pure waste.

Build the client once, share it everywhere

A reqwest::Client is expensive to create but cheap to clone. Cloning shares the same connection pool and configuration. Create it at startup and pass Arc<Client> to workers, HTTP handlers, or background jobs.

use std::sync::Arc;
use reqwest::Client;

fn build_client() -> Arc<Client> {
    let client = Client::builder()
        .pool_max_idle_per_host(32)
        .tcp_keepalive(std::time::Duration::from_secs(60))
        .build()
        .expect("client build");
    Arc::new(client)
}

If you construct Client::new() inside a request handler, you throw away the pool on every call. Don’t. In an Actix or Axum app, store the client in app state:

// Axum example
use axum::extract::State;
struct AppState { client: Arc<Client> }
async fn handler(State(state): State<Arc<AppState>>) -> String {
    let _ = state.client.get("https://api.example.com/health").send().await;
    "ok".into()
}

Tune the pool for concurrency

Set pool_max_idle_per_host to match your typical concurrent requests to a single endpoint. For a service that sends 50 parallel chat completions to one model host, keep at least that many idle connections ready.

let client = Client::builder()
    .pool_max_idle_per_host(64)
    .pool_idle_timeout(std::time::Duration::from_secs(30))
    .build()?;

pool_idle_timeout controls how long an idle connection lives. Too short and you lose reuse; too long and you hold dead sockets when the server closes them. 30 s is a sane default; many LLM gateways use 60–90 s keep-alive. Set tcp_keepalive to slightly below that to detect dead peers.

Streaming changes the math

Streaming responses hold the connection open until the final token. If you run 100 concurrent streams, you need 100 active connections, not just idle ones. The pool limits idle, not active, but active connections return to idle after the stream ends. Ensure your runtime has enough file descriptors: ulimit -n matters.

let resp = client
    .post("https://api.example.com/v1/chat/completions")
    .json(&serde_json::json!({
        "model": "gpt-4o",
        "messages": [{"role": "user", "content": "hi"}],
        "stream": true
    }))
    .send()
    .await?;

let mut stream = resp.bytes_stream();
use futures::StreamExt;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    // process token
}
// connection returns to pool here

Timeouts and cancellation

A pooled connection that hangs wastes a slot. Set a client-wide timeout only if your total request budget is fixed. For streaming, wrap consumption in tokio::time::timeout so you drop the response and free the connection.

use tokio::time::{timeout, Duration};
let guarded = timeout(Duration::from_secs(30), async {
    let mut stream = resp.bytes_stream();
    while let Some(c) = stream.next().await { /* ... */ }
}).await;

Dropping the reqwest::Response mid-stream closes the connection (it does not return to pool). That’s correct: a half-read body is unsafe to reuse. If you abort a stream, expect a new connection next time.

One endpoint, many models

If you call multiple model providers directly, each host gets its own pool partition. That’s fine, but you now manage N pools. An inference gateway such as n4n.ai exposes a single OpenAI-compatible endpoint for 240+ models; pointing your pooled client there means one pool serves every model switch, maximizing reuse without code changes.

Common pitfalls

• New client per task: Client::new() in a loop kills pooling. Clone the shared Arc<Client>.
• Default pool size: Forgetting pool_max_idle_per_host leaves you at 1 idle connection. Set it explicitly.
• HTTPS proxy interference: If you route through a corporate proxy, the proxy may terminate keep-alive. Test end-to-end.
• DNS rotation: If a host resolves to multiple IPs, hyper pools per (host, port) not per IP, but the OS may hand out different IPs. Use a stable endpoint or a gateway.
• Not setting tcp_keepalive: Without it, idle connections may be silently dropped by NAT, causing reused-socket errors. Reqwest enables TCP_NODELAY by default; keepalive is optional but recommended.
• Blocking client in async: reqwest::blocking::Client has its own pool but cannot be used inside async runtimes without spawning threads. Use async client for tokio services.

Ordered implementation path

1. Create a Client at process start with pool_max_idle_per_host sized to your concurrency.
2. Wrap it in Arc and inject into services or app state.
3. Use .post().json() for chat completions; reuse the same client for all models.
4. For streaming, consume the full body or explicitly drop on timeout.
5. Set tcp_keepalive and pool_idle_timeout to values just below your server’s keep-alive window.
6. Monitor open connections with ss -tnp or metrics from your proxy.

Measuring success

You can verify pooling works by checking connection counts during a load test:

ss -tn 'sport = :443' | wc -l

If the number stabilizes near your concurrency limit instead of growing unbounded, pooling is engaged. Enable RUST_LOG=reqwest=debug to see pool hits:

RUST_LOG=reqwest=debug cargo run

You’ll see internal hyper logs about connection reuse. If you see constant connecting events, the pool is misconfigured.

Tradeoffs

A larger pool uses more memory and file descriptors. On a 1k-concurrent stream server, 1k idle connections is fine; on a tiny lambda, keep it small. Connection pooling does not help cold starts—the first call always pays the handshake. For serverless Rust, consider keeping a warm client across invocations via a global OnceLock.

use std::sync::OnceLock;
static CLIENT: OnceLock<Client> = OnceLock::new();
fn client() -> &'static Client {
    CLIENT.get_or_init(|| Client::builder().pool_max_idle_per_host(16).build().unwrap())
}

Final notes on client directives

When you send Cache-Control or routing hints, reqwest forwards them untouched. The pool is agnostic to request headers; only the host matters. If you need per-model isolation (e.g., separate rate limits), use different Client instances or distinct hostnames. Rust reqwest connection pooling is a baseline optimization that pays for itself on day one; configure it deliberately and your LLM client will scale without melting the network stack.