Calling large language model endpoints from PHP demands resilience. A well-configured guzzle retry middleware llm api layer absorbs transient 429s and 5xxs so your application doesn’t bubble up noise. This guide walks through building that middleware from scratch and wiring it into a real client.
Step 1: Identify which failures warrant a retry
LLM inference is a network call to a stateless HTTP service. The failures you can safely retry fall into two buckets: server-side transient errors and client-side transport errors.
Server-side: HTTP 429 (rate limited), 500 (internal error), 502/503/504 (gateway or overload). These are emitted by the model provider or your inference gateway. A 429 usually includes a Retry-After header; respect it.
Transport errors happen below HTTP: connection refused, TLS handshake reset, cURL timeout (code 28), or empty reply from server. Guzzle surfaces these as RequestException with a cURL error code.
Do not retry 400, 401, 403, or 404. Those are deterministic and indicate a bug in your request shape or credentials, not transient load. A guzzle retry middleware llm api implementation must encode this distinction or you’ll mask real defects.
Step 2: Write a retry decision function
Separate the “should retry” logic from the retry loop. This keeps the middleware readable and unit-testable.
<?php
use GuzzleHttp\Exception\RequestException;
use Psr\Http\Message\ResponseInterface;
function isRetryable(int $retries, ?ResponseInterface $response, ?\Throwable $e): bool
{
if ($retries >= 5) {
return false; // hard cap
}
if ($response instanceof ResponseInterface) {
$status = $response->getStatusCode();
if ($status === 429 || $status >= 500) {
return true;
}
}
if ($e instanceof RequestException) {
// cURL codes: 7 = connection refused, 28 = timeout, 52 = empty reply
$curlCode = $e->getCode();
if (in_array($curlCode, [7, 28, 52], true)) {
return true;
}
}
return false;
}This function returns true only for the retryable classes above. The $retries argument lets you enforce a maximum attempt count centrally.
Step 3: Build the guzzle retry middleware llm api stack with backoff
Guzzle ships Middleware::retry(), which accepts a decision callable and a delay callable. The delay should be exponential with jitter to avoid thundering herds.
<?php
use GuzzleHttp\Middleware;
use GuzzleHttp\HandlerStack;
$stack = HandlerStack::create();
$stack->push(Middleware::retry(
function (int $retries, $request, $response, $exception) {
return isRetryable($retries, $response, $exception);
},
function (int $retries, $response) {
// Exponential backoff: 200ms, 400ms, 800ms...
$base = 200 * (2 ** $retries);
// Add up to 100ms jitter
return $base + random_int(0, 100);
}
));If you need finer control—for example, honoring Retry-After on 429s—replace the delay callable:
function delayFromResponse(int $retries, ?ResponseInterface $response): int
{
if ($response && $response->getStatusCode() === 429) {
$header = $response->getHeaderLine('Retry-After');
if (is_numeric($header)) {
return (int) ($header * 1000); // seconds to ms
}
}
return 200 * (2 ** $retries) + random_int(0, 100);
}Push that into the middleware instead of the inline closure.
Step 4: Handle idempotency and non-safe methods
POST to /v1/chat/completions is not an HTTP safe method, but inference is side-effect-free if the provider doesn’t log or bill twice. Most LLM gateways bill per token on the successful response, so a duplicate POST that yields two completions costs double.
Mitigate by generating an idempotency key per logical request and sending it as a header. If your gateway supports it (some OpenAI-compatible ones do), it will dedupe.
<?php
$key = bin2hex(random_bytes(16));
$request = $request->withHeader('Idempotency-Key', $key);Only attach the key once per logical operation, not per retry. Reuse the same key across attempts so the server can correlate them.
Step 5: Configure the Guzzle client
Instantiate the client with the handler stack and sane timeouts. LLM calls can take tens of seconds; set timeout high enough to avoid false retries on slow generation.
<?php
use GuzzleHttp\Client;
$client = new Client([
'handler' => $stack,
'base_uri' => 'https://api.example-llm.com',
'timeout' => 90,
'connect_timeout' => 10,
'headers' => [
'Authorization' => 'Bearer ' . getenv('LLM_API_KEY'),
'Content-Type' => 'application/json',
],
]);
$response = $client->post('/v1/chat/completions', [
'json' => [
'model' => 'gpt-4o-mini',
'messages' => [['role' => 'user', 'content' => 'Hello']],
],
]);The guzzle retry middleware llm api configuration now wraps every call made through $client.
Step 6: Account for LLM streaming and gateway behavior
Streaming responses (stream => true in Guzzle) break naive retries. If you get a 503 after the headers but before the first token, the body promise may have partially emitted. Guzzle’s Middleware::retry rejects the promise and retries, but your application may have already sent bytes to the browser.
For streaming, either disable retries or buffer the first chunk before forwarding:
$stack->push(Middleware::retry(
function ($retries, $request, $response, $exception) {
if ($request->getHeaderLine('X-Stream') === 'true') {
return false; // don't retry streams
}
return isRetryable($retries, $response, $exception);
},
fn($retries) => 200 * (2 ** $retries)
));If you route through a gateway such as n4n.ai, its single OpenAI-compatible endpoint already performs automatic fallback when a upstream provider is rate-limited or degraded. Client-side retries still matter for TCP resets and local network blips, but you can lower your max retries because the gateway absorbs provider-level failures.
Step 7: Verify with a local mock server
Don’t trust the middleware until you watch it retry. Stand up a tiny PHP mock that fails twice then succeeds.
<?php
// mock.php
session_start();
$attempt = $_SESSION['attempt'] ?? 0;
$_SESSION['attempt'] = $attempt + 1;
if ($attempt < 2) {
http_response_code(503);
header('Content-Type: application/json');
echo json_encode(['error' => 'degraded']);
exit;
}
http_response_code(200);
header('Content-Type: application/json');
echo json_encode(['ok' => true, 'attempt' => $attempt]);Run it:
php -S localhost:8001 mock.phpPoint a test client at it with the retry stack and assert you get the 200 after two 503s:
<?php
$stack = HandlerStack::create();
$stack->push(Middleware::retry(
fn($retries, $req, $resp, $ex) => isRetryable($retries, $resp, $ex),
fn($retries) => 50 * (2 ** $retries)
));
$testClient = new Client(['handler' => $stack, 'base_uri' => 'http://localhost:8001']);
$resp = $testClient->get('/');
assert($resp->getStatusCode() === 200);
echo "Retries succeeded: " . $resp->getBody()->getContents() . "\n";You should see the final JSON with attempt: 2. If you kill the mock entirely, the cURL timeout path should trigger retries and then surface a RequestException after the cap.
Step 8: Log and meter what you retry
Retries hide latency. Emit a structured log on each retry with the attempt count and status so you can spot a degraded dependency.
$stack->push(Middleware::retry(
function ($retries, $request, $response, $exception) use ($logger) {
$should = isRetryable($retries, $response, $exception);
if ($should && $retries > 0) {
$logger->warning('Retrying LLM call', [
'attempt' => $retries,
'uri' => (string) $request->getUri(),
'status' => $response?->getStatusCode(),
]);
}
return $should;
},
fn($retries) => 200 * (2 ** $retries)
));If you’re on a per-token metering plan, correlate retry attempts with request IDs from the gateway response to avoid paying for duplicated generations.
A guzzle retry middleware llm api layer is not optional for production PHP services. Build the decision function, cap the attempts, back off exponentially, and verify against a mock before you ship.