Rate limits are the tax you pay for shared infrastructure. When you build a Go client for an LLM gateway, implementing go exponential backoff rate limit logic is the difference between a resilient service and a retry storm that gets you blocked. This guide walks through wrapping net/http with a retry transport that backs off on 429 and 503, honors Retry-After, and adds jitter so your clients don’t synchronize into a thundering herd.
Step 1: Classify retryable responses and errors
Not every failure should trigger a retry. A 400 means your request is malformed; a 429 means the server is asking you to slow down; a 503 means it is temporarily overloaded. Network-level errors (connection reset, timeout, DNS failure) are usually transient. Start with an explicit classifier so you never accidentally retry a permanent error.
func isRetryable(status int, err error) bool {
if err != nil {
// Treat transient network errors as retryable.
return true
}
switch status {
case http.StatusTooManyRequests, // 429
http.StatusServiceUnavailable: // 503
return true
}
return false
}If the upstream sends X-RateLimit-Remaining: 0 but still returns 200, that is not a retry trigger—it is a budgeting signal for your own caller. Only retry on explicit failure signals.
Step 2: Compute exponential backoff with a sane cap
Pure exponential growth explodes: 1s, 2s, 4s, 8s, 16s, 32s. For LLM inference calls that typically complete in seconds to minutes, a 30s hard cap is reasonable. Use a base that matches provider reset windows (often 500ms–2s).
type Backoff struct {
Base time.Duration
Max time.Duration
Factor float64
}
func (b Backoff) ForAttempt(attempt int) time.Duration {
if attempt <= 0 {
return b.Base
}
d := float64(b.Base) * math.Pow(b.Factor, float64(attempt))
if d > float64(b.Max) {
return b.Max
}
return time.Duration(d)
}A Base of 500ms, Factor of 2.0, and Max of 30s avoids both aggressive hammering and absurd waits. The go exponential backoff rate limit pattern only works if the cap reflects your latency budget.
Step 3: Add jitter to break synchronization
If every client computes the same delay, they all retry simultaneously after an outage. Full jitter randomizes the delay between zero and the computed cap. Decorrelated jitter is another option, but full jitter is simplest and effective.
func (b Backoff) ForAttemptJittered(attempt int, r *rand.Rand) time.Duration {
d := b.ForAttempt(attempt)
return time.Duration(r.Float64() * float64(d))
}Seed a local rand.Rand with rand.New(rand.NewSource(...)). Do not use the global rand functions in high-throughput paths—they contend on a shared lock.
Step 4: Honor Retry-After from the server
A 429 or 503 often includes a Retry-After header. It may be an integer number of seconds or an HTTP-date. When present, it overrides your computed backoff because the server knows its own window.
func retryAfter(res *http.Response, computed time.Duration) time.Duration {
if res == nil {
return computed
}
ra := res.Header.Get("Retry-After")
if ra == "" {
return computed
}
if secs, err := strconv.Atoi(ra); err == nil {
return time.Duration(secs) * time.Second
}
if t, err := http.ParseTime(ra); err == nil {
if d := time.Until(t); d > 0 {
return d
}
}
return computed
}If the header specifies a date in the past or zero seconds, fall back to your jittered value.
Step 5: Wrap net/http with a retry RoundTripper
The cleanest integration is a custom http.RoundTripper. It delegates to the underlying transport and retries on retryable outcomes. This leaves your http.Client usage unchanged.
type retryTransport struct {
rt http.RoundTripper
backoff Backoff
rand *rand.Rand
maxAttempts int
}
func (t *retryTransport) RoundTrip(req *http.Request) (*http.Response, error) {
var lastErr error
for attempt := 0; attempt < t.maxAttempts; attempt++ {
// Rebuild body if the request supports it.
r := req.Clone(req.Context())
if req.GetBody != nil {
body, err := req.GetBody()
if err != nil {
return nil, err
}
r.Body = body
}
res, err := t.rt.RoundTrip(r)
if err == nil && !isRetryable(res.StatusCode, nil) {
return res, nil
}
if res != nil {
res.Body.Close()
}
lastErr = err
if attempt == t.maxAttempts-1 {
break
}
delay := t.backoff.ForAttemptJittered(attempt, t.rand)
if res != nil {
delay = retryAfter(res, delay)
}
select {
case <-time.After(delay):
case <-req.Context().Done():
return nil, req.Context().Err()
}
}
if lastErr != nil {
return nil, lastErr
}
return nil, errors.New("retry failed: max attempts reached")
}req.GetBody is populated automatically for requests created via http.NewRequest with a bytes.Reader or strings.Reader. If you build requests manually, set GetBody yourself.
Step 6: Make POST retries safe with idempotency
LLM completion calls are POSTs. Blindly retrying them can double-bill or produce duplicate generations. Either:
- Send an
Idempotency-Keyheader and ensure your gateway honors it. - Buffer the request body and seek it back to start before each retry.
func newRetryablePost(url string, body []byte) (*http.Request, error) {
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return nil, err
}
// GetBody is set automatically by NewRequest for bytes.Reader.
req.Header.Set("Idempotency-Key", uuid.NewString())
return req, nil
}If you route through n4n.ai, its OpenAI-compatible endpoint honors client routing directives and forwards provider cache-control hints, but idempotency is still your application’s responsibility. The go exponential backoff rate limit wrapper must not introduce side effects.
Step 7: Build a minimal OpenAI-compatible client
Wire the transport into an http.Client and call a chat endpoint. Use a context with a timeout larger than your max backoff sum.
client := &http.Client{
Transport: &retryTransport{
rt: http.DefaultTransport,
backoff: Backoff{Base: 500 * time.Millisecond, Max: 30 * time.Second, Factor: 2},
rand: rand.New(rand.NewSource(time.Now().UnixNano())),
maxAttempts: 5,
},
Timeout: 2 * time.Minute,
}
payload := map[string]any{
"model": "gpt-4o-mini",
"messages": []map[string]string{
{"role": "user", "content": "Say hi"},
},
}
b, _ := json.Marshal(payload)
req, _ := newRetryablePost("https://api.example.com/v1/chat/completions", b)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+os.Getenv("API_KEY"))
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// decode respThis client now survives transient provider 429s without manual intervention.
Step 8: Verify with a local mock server
Use httptest to simulate an upstream that returns 429 twice (with Retry-After: 0) then 200. This proves the retry loop executes the right number of times.
func TestRetry(t *testing.T) {
var hits int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits++
if hits <= 2 {
w.Header().Set("Retry-After", "0")
w.WriteHeader(http.StatusTooManyRequests)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
c := &http.Client{Transport: &retryTransport{
rt: srv.Client().Transport,
backoff: Backoff{Base: time.Millisecond, Max: 10 * time.Millisecond, Factor: 2},
rand: rand.New(rand.NewSource(1)),
maxAttempts: 5,
}}
res, err := c.Get(srv.URL)
if err != nil || res.StatusCode != 200 {
t.Fatalf("expected 200, got %v %v", res.StatusCode, err)
}
if hits != 3 {
t.Fatalf("expected 3 hits, got %d", hits)
}
}Run go test ./.... A pass means your go exponential backoff rate limit transport retried exactly twice and succeeded on the third attempt.
Verify success in production
In production, emit a counter for retry_attempts and a gauge for final_status. Log the attempt index and any Retry-After received. A healthy integration shows retry rates under 1% during steady load. If retries climb, your base delay is too small or the provider is saturated—shard your API keys or move to a different model tier.
The implementation above is minimal but production-grade. It respects server hints, avoids synchronized retries, and keeps your LLM client boring—which is the only correct goal for error handling.