Connection pooling for high-throughput LLM calls in Go

If you’re shipping a service that fans out prompts to a model endpoint, the stock http.DefaultClient will bottleneck you before the provider does. Getting go http.client connection pooling llm tuning right is the difference between saturating your rate limit and burning milliseconds on redundant TCP/TLS handshakes. This guide walks through the exact transport settings and concurrency patterns we use for high-throughput inference calls from Go.

Profile before you tune

Don’t guess. Point a load test at your target endpoint with hey or a custom Go benchmark and watch net/http trace metrics. The key signal is connection reused vs connection established in http.Transport logs (enable via GODEBUG=http2debug=2 or a custom ClientTrace). If you see few reuses, your pool is too small or idle connections are expiring prematurely.

A minimal trace hook:

trace := &httptrace.ClientTrace{
    GotConn: func(info httptrace.GotConnInfo) {
        if !info.Reused {
            // count cold connects
        }
    },
}

Cold connects add 20–100ms depending on TLS. At 500 QPS that tax dominates tail latency.

The http.Client is safe for concurrent use and is designed to be long-lived. Never create a client per request. The client holds the Transport, which owns the connection pool. A new client per call means a new pool per call—defeating go http.client connection pooling llm entirely.

var llmClient = newLLMClient() // package-level singleton

func newLLMClient() *http.Client {
    // see transport config below
}

If you need different timeouts for streaming vs non-streaming, use two clients with distinct transports, but still reuse each.

Configure the Transport, not just the Client

The Timeout field on http.Client is a hard stop for the entire exchange. Connection pooling lives in http.Transport. The defaults are hostile to high throughput: MaxIdleConnsPerHost is 2. That means regardless of how many goroutines you spawn, only two idle connections per host are kept alive. Everything else closes after the response.

Set these explicitly:

transport := &http.Transport{
    MaxIdleConns:        500,
    MaxIdleConnsPerHost: 500,
    MaxConnsPerHost:     500,
    IdleConnTimeout:     90 * time.Second,
    TLSHandshakeTimeout: 10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}

MaxConnsPerHost caps total open connections (idle + active). If you exceed it, Transport blocks until a connection frees. Size it to your rate limit and goroutine count.

Size the pool to your concurrency

A simple formula: pool_size >= expected_concurrent_requests. If your worker pool fires 200 concurrent LLM calls and each call takes 800ms, you need at least 200 connections to avoid serialization. Add headroom for retries and health checks.

But don’t set it to 100000 blindly. Each idle connection consumes a file descriptor and some memory. On Linux, default per-process FD limit is often 1024. Raise ulimit -n or set MaxIdleConns conservatively. We typically run 500–2000 per host on services with 8–16k FD headroom.

Idle timeouts and HTTP/2

LLM endpoints usually support HTTP/2. With h2, a single TCP connection multiplexes many streams, so MaxIdleConnsPerHost matters less—but MaxConnsPerHost still bounds concurrency. Go’s Transport uses h2 automatically when the server advertises it.

Set IdleConnTimeout to slightly less than the server’s keep-alive (commonly 60–120s). If you leave it too long, you’ll hold dead connections after a load balancer silently drops them, causing write: broken pipe on reuse. We use 90s.

// force HTTP/2 if needed (usually automatic)
transport.ForceAttemptHTTP2 = true

Use context for cancellation, not client timeout

Streaming LLM responses can run long. A fixed Client.Timeout will kill a healthy stream. Instead, pass a context.Context with the request and rely on context.WithTimeout or WithDeadline per call. The transport cancels the request and returns the connection to the pool faster.

req, _ := http.NewRequestWithContext(ctx, "POST", url, body)
resp, err := llmClient.Do(req)

For non-streaming chat completions, a 30s client timeout is fine. For streaming, set Timeout: 0 on the client and manage via context.

Beware retry storms and auth headers

When a provider returns 429 or 503, your code likely retries. If you retry by spawning new goroutines without bounding, you can exhaust the pool and trigger a connection storm. Use a golang.org/x/sync/semaphore or buffered channel to cap in-flight calls.

Also, if you sign requests with a per-request token that changes, connection coalescing still works (TLS session resumption is independent of auth). But if you use Proxy-Authorization or mutate Transport per request, you break the pool. Keep the transport immutable.

A gateway like n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint and handles automatic fallback when a provider is degraded; your Go client should pool against that single endpoint exactly as above, letting the gateway manage routing.

Monitor and adjust

Export transport stats via a debug handler:

import "net/http/pprof"
// register debug endpoints, then scrape /debug/pprof/heap and custom metrics

Track idle_conns, active_conns, and wait_for_conn (the latter appears as latency spikes when MaxConnsPerHost is hit). If wait_for_conn climbs, raise limits or shed load.

Tradeoffs summary

Larger pools: lower latency, higher FD/memory cost.
Smaller pools: bounded resources, risk of queuing.
HTTP/2: fewer connections, but head-of-line blocking on a single TCP if packet loss occurs.
Client per request: never do it.

Tune go http.client connection pooling llm against real traffic, not intuition. The settings above are a baseline we ship; your rate limits and payload sizes will shift the numbers.