Timeout tuning for LLM API requests: best practices

Most teams copy a 30-second timeout from a REST tutorial and call it done. That ignores the latency profile of generative endpoints, where token streaming and provider load swings make naive limits either abort good requests or hang threads. Following llm api request timeout best practices means separating connect from read timeouts, budgeting for output length, and designing retries that don’t amplify load.

1. Measure the real latency distribution first

You cannot tune a timeout without data. LLM endpoints exhibit a heavy-tailed time-to-first-token (TTFT) and a linear generation phase driven by output token count. A 100-token response and a 4K-token response from the same model differ by an order of magnitude in wall-clock time.

Run a sample batch against your target model and record percentiles:

curl -s -o /dev/null -w "connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
  -X POST https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"mistral-7b","messages":[{"role":"user","content":"Explain TCP slow start in 200 words"}]}'

Do this for both short and long max_tokens values. The p99 TTFT tells you the floor for any read timeout; the generation rate (tokens/sec) tells you the slope.

2. Split connect and read timeouts

A single total timeout is the most common mistake. TCP connection establishment and TLS handshake fail fast or hang long depending on DNS and network path. Generation read time should be independent.

In Python with httpx (which the OpenAI SDK uses under the hood):

from openai import OpenAI

client = OpenAI(
    base_url="https://api.example.com/v1",
    api_key="sk-...",
    timeout=(3.0, 60.0),  # (connect, read)
)

The connect timeout of 3 seconds catches a dead upstream or misrouted DNS. The read timeout of 60 seconds covers TTFT plus generation for your expected output size. If you pass a single float, both are equal—bad for LLMs.

Pitfall: read timeout as total time

Some HTTP clients treat read timeout as inactivity between bytes, others as total request duration. Verify your stack. httpx uses it as inactivity per read chunk, which is what you want for streaming.

3. Budget read timeout per output token

Once you know the generation rate, compute the read budget:

read_timeout = p99_ttft + (max_tokens / min_observed_tps) * safety_margin

If p99 TTFT is 2s, you request 2,000 tokens, and the slowest observed rate is 20 tps, budget 2 + (2000/20)*1.5 = 152 seconds. Round to 160. Hard-coding 30s will kill 90% of those requests.

This is core to llm api request timeout best practices: the limit is a function of your own request parameters, not a universal constant.

4. Stream and enforce idle timeouts

Non-streaming calls force you to wait for the full payload, making a single read timeout the only defense. Streaming gives you byte-level liveness. Configure the client to abort if no chunk arrives within N seconds, even if total time is large.

import httpx

transport = httpx.HTTPTransport(retries=0)
client = httpx.Client(timeout=httpx.Timeout(3.0, read=10.0), transport=transport)

with client.stream("POST", url, json=payload, headers=headers) as r:
    for line in r.iter_lines():
        if line.startswith("data:"):
            # process token

Here read=10.0 means “no more than 10s between chunks.” A stalled provider triggers a clean timeout instead of a silent hang.

Tradeoff: partial output

On timeout mid-stream, you already have tokens. Decide whether to surface partial text or discard. For chat UX, showing partial with a “generation interrupted” note often beats an error.

5. Retries with backoff, jitter, and scopes

Timeouts and 429/5xx are retryable. 400/401 are not. Retry only the former, with exponential backoff and full jitter to avoid thundering herds.

TypeScript example with explicit abort per attempt:

async function callWithRetry(url: string, opts: RequestInit, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    const ctrl = new AbortController();
    const timer = setTimeout(() => ctrl.abort(), 8000); // per-attempt read cap
    try {
      const res = await fetch(url, { ...opts, signal: ctrl.signal });
      clearTimeout(timer);
      if (res.status === 429 || res.status >= 500) throw new Error("retryable");
      return res;
    } catch (e) {
      clearTimeout(timer);
      if (i === attempts - 1) throw e;
      const backoff = Math.min(1000 * 2 ** i, 10000) + Math.random() * 500;
      await new Promise((r) => setTimeout(r, backoff));
    }
  }
  throw new Error("unreachable");
}

Pitfall: retrying non-idempotent generations

A timeout after the provider generated tokens but before response delivery means a retry may produce a second, different completion. If your system charges per call or writes to a log, you get duplicates. Pass a client-generated request_id if the gateway supports dedupe, or accept the cost.

6. Gateway and fallback behavior

If you sit behind an inference gateway such as n4n.ai, automatic provider fallback changes the latency tail but not the need for client timeouts; set read timeout to cover the slowest routed model plus fallback overhead. A gateway that honors routing directives and forwards cache-control hints can reduce TTFT via provider-side caching, but your connect timeout still must catch a dead edge node.

When a primary provider is rate-limited, the gateway shifts to a secondary. That handoff adds latency. Measure p99 with fallback enabled before fixing your limits.

7. Monitor and tune continuously

Timeouts are not set-and-forget. Export two metrics:

• timeout_rate by model and endpoint
• effective_latency_p99 excluding timed-out requests

If timeout_rate exceeds 1%, your read budget is too tight for the current provider health. If it is near zero but effective_latency_p99 is far below your limit, you are wasting capacity holding connections open—tighten connect timeout and consider lower read cap with streaming.

{
  "metric": "llm_timeout_rate",
  "tags": {"model": "mixtral-8x7b", "stream": true},
  "value": 0.004,
  "p99_latency_ms": 42000
}

Alert on sudden jumps. Provider degradation often shows as TTFT inflation hours before full outages.

8. Common pitfalls summary

• Single timeout value: Kills long generations or hides network faults.
• Retrying on client abort only: Forgets to retry on 429 from upstream proxy.
• No idle timeout on stream: A stalled connection holds a worker indefinitely.
• Ignoring token budget: Using same timeout for 50-token and 4K-token calls.
• Synchronous blocking calls: In async services, a 120s timeout blocks an event loop thread if not awaited properly.

9. Ordered checklist

1. Profile TTFT p99 and tokens/sec for each model you call.
2. Set connect timeout ≤ 5s, read timeout = TTFT p99 + token budget × 1.5.
3. Enable streaming; set read timeout as idle-per-chunk.
4. Implement retry only for timeout/429/5xx with capped exponential backoff + jitter.
5. Pass request IDs if gateway supports dedupe to avoid double generation.
6. Monitor timeout rate and p99; adjust monthly or after provider incidents.

These llm api request timeout best practices keep your services resilient without masking real provider latency. The goal is to fail fast on dead paths and patiently wait on live ones—not to pick a number and hope.