Python requests session pooling for high-volume LLM calls

If you fire thousands of completions per minute at an inference endpoint, the overhead of opening a new TCP/TLS session per call will dominate your latency budget. Implementing python requests session pooling llm clients keeps connections alive and amortizes TLS handshakes across many requests, which is the difference between 30 ms and 300 ms tail latencies in practice.

Step 1: Stop calling requests.post in a loop

The naive pattern engineers reach for is requests.post(...) inside a loop. Each call constructs a fresh Session implicitly, performs a DNS lookup, a TCP three-way handshake, and a full TLS negotiation. At 10 req/s that’s 20 extra round trips per second of pure overhead.

# Bad: new connection every iteration
import requests

def complete(prompt):
    return requests.post(
        "https://api.example.com/v1/chat/completions",
        json={"model": "gpt-4o", "messages": [{"role": "user", "content": prompt}]},
        headers={"Authorization": "Bearer KEY"}
    ).json()

That code works for a cron job. It falls over at volume.

Step 2: Create a module-level Session with a sized pool

A requests.Session reuses underlying TCP connections via urllib3 connection pools. You control the pool size through an HTTPAdapter. Mount it for both http:// and https://.

import requests
from requests.adapters import HTTPAdapter

def make_session(pool_size: int = 100) -> requests.Session:
    session = requests.Session()
    adapter = HTTPAdapter(
        pool_connections=pool_size,
        pool_maxsize=pool_size,
        max_retries=0  # we handle retries explicitly
    )
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

session = make_session(200)

pool_connections is the number of separate host pools to keep; pool_maxsize is the max connections per host. For a single LLM gateway, set both to the same value equal to your peak concurrent requests. If you exceed it, threads block waiting for a free connection—better than spawning sockets, but size it correctly.

Step 3: Attach explicit timeouts and retry policy

Never call a remote LLM without a timeout. Defaults are indefinite. Use urllib3.util.retry.Retry for idempotent failures, but POST completions are not safely retried blindly because you may get charged twice. Use short connect/read timeouts and a custom retry only on connection errors and 429/503.

from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def make_resilient_session(pool_size: int = 100) -> requests.Session:
    session = requests.Session()
    retry = Retry(
        total=3,
        connect=3,
        read=1,
        status=1,
        status_forcelist=[429, 500, 502, 503],
        backoff_factor=0.5,
        allowed_methods=frozenset(["GET", "POST"])  # only retry if you accept dupes
    )
    adapter = HTTPAdapter(pool_connections=pool_size, pool_maxsize=pool_size, max_retries=retry)
    session.mount("https://", adapter)
    return session

Set per-request timeout:

resp = session.post(url, json=payload, headers=headers, timeout=(3.05, 30))

Connect timeout 3.05 s, read timeout 30 s. Tune read to your model’s generation length.

Step 4: Write a completion helper that uses the pool

Keep the session global and pass it around. The JSON shape below matches the OpenAI compatibility spec used by most gateways.

import os

API_KEY = os.environ["LLM_API_KEY"]
BASE_URL = os.environ.get("LLM_BASE_URL", "https://api.example.com/v1")

def chat_completion(session, model: str, prompt: str, max_tokens: int = 512):
    url = f"{BASE_URL}/chat/completions"
    headers = {"Authorization": f"Bearer {API_KEY}"}
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": False,
    }
    resp = session.post(url, json=payload, headers=headers, timeout=(3.05, 60))
    resp.raise_for_status()
    return resp.json()

When targeting n4n.ai, the single OpenAI-compatible endpoint addresses 240+ models and forwards provider cache-control hints, so you can swap model in the payload without reconstructing the session or changing BASE_URL.

Step 5: Drive concurrency with ThreadPoolExecutor

requests is blocking, so use threads to saturate the pool. Your worker count must be ≤ pool_maxsize, or threads will queue.

from concurrent.futures import ThreadPoolExecutor, as_completed

def run_batch(session, prompts, model="gpt-4o-mini", workers=50):
    results = []
    with ThreadPoolExecutor(max_workers=workers) as ex:
        futures = [ex.submit(chat_completion, session, model, p) for p in prompts]
        for f in as_completed(futures):
            try:
                results.append(f.result())
            except Exception as e:
                results.append({"error": str(e)})
    return results

prompts = [f"Summarize item {i}" for i in range(500)]
sess = make_resilient_session(pool_size=50)
out = run_batch(sess, prompts, workers=50)

If you need more than a few hundred concurrent calls, move to httpx with asyncio and limits.MaxConnections. The pooling concept is identical; only the event loop changes.

Step 6: Verify connection reuse under load

Success is not “it returns JSON.” You must confirm sockets are reused. Two quick checks:

1. Host connection count. While the batch runs, inspect established connections to your gateway IP:

   ss -tnp | grep ':443' | wc -l

   With pool_size=50 and 50 workers, the count should hover near 50, not 500.

2. Latency delta. Time a cold loop (new requests.post per call) vs the pooled session for 200 calls. The pooled version should show dramatically lower p95 after the first few requests as the pool warms.

import time

def bench(session, prompts):
    t0 = time.perf_counter()
    run_batch(session, prompts, workers=50)
    return time.perf_counter() - t0

print(f"pooled: {bench(sess, prompts):.2f}s")

If you see connection counts climbing linearly with request count, the session is not shared—likely you rebuilt it inside the worker function.

Step 7: Handle pool exhaustion and backpressure

When pool_maxsize is smaller than demand, urllib3 blocks the calling thread until a connection frees. That is correct behavior, but you must surface it. Wrap submissions with a semaphore if you want to bound memory:

import threading

sem = threading.Semaphore(50)

def bounded_submit(ex, session, prompt):
    with sem:
        return chat_completion(session, "gpt-4o-mini", prompt)

Also set session.keep_alive = True (default) and ensure your gateway sends Connection: keep-alive. Some load balancers terminate idle connections after 60 s; if you see sporadic ConnectionError, lower pool_maxsize or add a Retry on connection errors as in Step 3.

Step 8: Clean shutdown

Sessions hold file descriptors. In a long-running service, close the session on shutdown:

import atexit
atexit.register(lambda: sess.close())

In serverless (Lambda, Cloud Run), the session dies with the instance; reuse it across invocations within the same warm container by storing it in a global variable, not inside the handler.

Common pitfalls

• Mounting only https:// while your BASE_URL is http:// silently creates a default adapter with pool size 10.
• Copying the session per thread. Session is thread-safe for requests; don’t clone it.
• Setting stream=True and not consuming the response leaks connections back to the pool only after the response object is closed. Always resp.close() or use with session.post(...) as r:.
• TLS session resumption helps but does not replace connection pooling; the TCP handshake still costs a round trip without a kept-alive socket.

When to move beyond requests

At 1k+ sustained QPS, Python threads hit GIL contention. Switch to httpx.AsyncClient with limits=httpx.Limits(max_connections=500, max_keepalive_connections=500). The pooling semantics map directly, and you get HTTP/2 multiplexing if the gateway supports it. Until that scale, a well-sized python requests session pooling llm client is the simplest robust solution.

Use the code above as a template. Measure, then adjust pool_maxsize to your concurrency, and keep timeouts strict. That’s the entire trick.