LLM providers fail in predictable ways: rate limits, transient 5xx, and dropped TCP connections. A go http.roundtripper retry llm api requests at the transport layer keeps retry logic out of your business code and works uniformly across any OpenAI-compatible SDK. Below is a complete implementation you can drop into a production client.
Step 1: Define the retry policy and transport wrapper
Start by declaring a struct that wraps an existing http.RoundTripper (usually http.DefaultTransport) and holds retry configuration. Keeping the policy in one place means every HTTP call your client makes gets the same treatment, whether it’s a chat completion, embedding, or audio transcript.
package retry
import (
"fmt"
"io"
"net/http"
"time"
)
type RetryTransport struct {
base http.RoundTripper
maxRetries int
backoff func(attempt int) time.Duration
}
func NewRetryTransport(base http.RoundTripper, maxRetries int) *RetryTransport {
if base == nil {
base = http.DefaultTransport
}
return &RetryTransport{
base: base,
maxRetries: maxRetries,
backoff: func(attempt int) time.Duration {
// exponential: 100ms, 200ms, 400ms, 800ms
return time.Duration(1<<uint(attempt)) * 100 * time.Millisecond
},
}
}The backoff function is exponential; tune the base and cap for your provider’s rate-limit headers. LLM endpoints often return Retry-After on 429s—we’ll honor that in Step 4. Avoid a naive fixed delay: providers that throttle aggressively will still be saturated if every client retries in lockstep.
Step 2: Implement the RoundTrip loop
The RoundTrip method must loop, call the base transport, and decide whether to retry. Only retry on network errors or on status codes 429, 500, 502, 503, 504. Never retry on 4xx other than 429—those are client errors and won’t fix themselves.
func (t *RetryTransport) shouldRetryStatus(resp *http.Response) bool {
if resp == nil {
return true // network error already handled by caller
}
switch resp.StatusCode {
case http.StatusTooManyRequests, http.StatusInternalServerError,
http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
return true
}
return false
}
func (t *RetryTransport) roundTripOnce(req *http.Request) (*http.Response, error) {
return t.base.RoundTrip(req)
}The full loop belongs in Step 3 after we solve body replay. The key design point: RoundTrip must be safe to call multiple times with the same *http.Request, which the standard library does not guarantee for request bodies.
Step 3: Buffer and replay the request body
http.Request.Body is a single-use stream. If you retry, the second RoundTrip gets an empty body and the LLM provider returns a 400. For LLM chat completions the payload is JSON, typically a few kilobytes. Read it into memory and reset the body each attempt.
func bufferBody(req *http.Request) ([]byte, error) {
if req.Body == nil {
return nil, nil
}
data, err := io.ReadAll(req.Body)
if err != nil {
return nil, err
}
req.Body.Close()
return data, nil
}
func (t *RetryTransport) RoundTrip(req *http.Request) (*http.Response, error) {
bodyBytes, err := bufferBody(req)
if err != nil {
return nil, err
}
var lastErr error
for attempt := 0; attempt <= t.maxRetries; attempt++ {
if bodyBytes != nil {
req.Body = io.NopCloser(bytes.NewReader(bodyBytes))
req.ContentLength = int64(len(bodyBytes))
req.GetBody = func() (io.ReadCloser, error) {
return io.NopCloser(bytes.NewReader(bodyBytes)), nil
}
}
if attempt > 0 {
delay := t.backoff(attempt - 1)
select {
case <-req.Context().Done():
return nil, req.Context().Err()
case <-time.After(delay):
}
}
resp, err := t.base.RoundTrip(req)
if err != nil {
lastErr = err
continue
}
if t.shouldRetryStatus(resp) {
resp.Body.Close()
lastErr = fmt.Errorf("retryable status %d", resp.StatusCode)
continue
}
return resp, nil
}
return nil, fmt.Errorf("retry failed after %d attempts: %w", t.maxRetries, lastErr)
}If you stream requests (rare for LLM inference), don’t buffer; instead require a GetBody function on the request, which the standard library supports natively. Setting GetBody as shown makes the request reusable even for SDKs that clone it.
Step 4: Honor context, deadlines, and Retry-After
LLM calls can take tens of seconds. Respect req.Context() for cancellation and use Retry-After if present. A 429 from an inference gateway often carries a precise wait time.
func retryAfter(resp *http.Response) time.Duration {
if resp == nil {
return 0
}
ra := resp.Header.Get("Retry-After")
if ra == "" {
return 0
}
if secs, err := strconv.Atoi(ra); err == nil {
return time.Duration(secs) * time.Second
}
if t, err := http.ParseTime(ra); err == nil {
return time.Until(t)
}
return 0
}In the loop, compute delay := max(t.backoff(attempt-1), retryAfter(resp)). Always select on ctx.Done() before sleeping so a cancelled request doesn’t burn a retry against a dead deadline.
Step 5: Wire into an OpenAI-compatible client
Most Go LLM SDKs accept an *http.Client. Set its Transport to your RetryTransport. This works for the official OpenAI Go module, any community client, or raw net/http.
client := &http.Client{
Transport: NewRetryTransport(http.DefaultTransport, 3),
Timeout: 60 * time.Second,
}
payload := []byte(`{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}`)
req, _ := http.NewRequestWithContext(ctx, "POST",
"https://api.example.com/v1/chat/completions", bytes.NewReader(payload))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)If you use a gateway such as n4n.ai, its OpenAI-compatible endpoint already performs automatic fallback when a provider is rate-limited or degraded, but the transport above is exactly the kind of logic you’d run when calling providers directly and want retries client-side.
Step 6: Verify success with fault injection
Write a test that fails the first two attempts with 503, then returns 200. Use httptest to confirm the transport retries and replays the body.
func TestRetryTransport(t *testing.T) {
var hits int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits++
body, _ := io.ReadAll(r.Body)
if len(body) == 0 && r.Method == "POST" {
t.Errorf("empty body on attempt %d", hits)
}
if hits < 3 {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
req, _ := http.NewRequest("POST", srv.URL, bytes.NewReader([]byte(`{}`)))
rt := NewRetryTransport(http.DefaultTransport, 3)
resp, err := rt.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
if resp.StatusCode != 200 || hits != 3 {
t.Fatalf("expected 200 after 3 hits, got %d hits status %d", hits, resp.StatusCode)
}
}Run go test ./.... You should see three server hits and a 200 response, with no empty-body errors. For production verification, point the client at a live endpoint and artificially throttle with a proxy, or check metrics for retry counts.
Step 7: Add jitter and observability
Pure exponential backoff causes thundering herds. Add full jitter: delay = rand.Duration(0, backoff). Also emit a metric per retry so you can see provider health.
import "math/rand"
func (t *RetryTransport) jitteredBackoff(attempt int) time.Duration {
base := t.backoff(attempt)
return time.Duration(rand.Int63n(int64(base)))
}Log or increment a counter when shouldRetryStatus is true. A go http.roundtripper retry llm api strategy is only complete when you can see retry rates per model and provider. Pair it with per-token metering to catch cost spikes from repeated attempts.
Caveats specific to LLM APIs
Retrying POST requests bills tokens on each attempt. Cap retries at 2–3 and only retry on clear transient signals. Streaming responses (SSE) complicate retries: if you get partial tokens, you can’t safely resume. Close the body and start over, but be aware the user may see duplicated text. For deterministic workloads, set a request ID and use provider-side idempotency if offered.
A go http.roundtripper retry llm api calls is a sharp tool. It removes boilerplate and centralizes resilience, but pair it with timeouts, circuit breakers, and sane backoff caps to keep both latency and costs under control.