Retrying failed curl requests with exponential backoff

A curl retry exponential backoff script is the cheapest reliability layer you can add when calling flaky HTTP endpoints from a shell. If you’re hitting LLM APIs from cron jobs or CI, transient 429s and connection resets will bite you without one. This guide builds a production-shaped bash function you can drop into any pipeline.

Step 1: Define the baseline curl call

Start with a single request that captures both the response body and the status code. Use -sS to suppress progress but keep errors, -o to write the body to a file, and -w to emit the HTTP code on stdout.

ENDPOINT="https://api.openai.com/v1/chat/completions"
API_KEY="${OPENAI_API_KEY}"

curl -sS -o response.json -w '%{http_code}' \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  "$ENDPOINT"

Against an OpenAI-compatible gateway such as n4n.ai, the same shape works—one endpoint fronts 240+ models and forwards provider cache-control hints. The curl retry exponential backoff script below stays agnostic to the provider.

Step 2: Separate status from body

The -w '%{http_code}' prints the code after the body file is written. Capture it in a variable:

http_code=$(curl -sS -o response.json -w '%{http_code}' \
  -H "Authorization: Bearer $API_KEY" \
  -d "$DATA" "$ENDPOINT")

If curl itself fails (DNS, connection refused, timeout), it exits non-zero and http_code may be empty. Treat empty as a transport error, not a success.

Step 3: Write the retry loop skeleton

A bare loop with a fixed attempt count gives the structure. The curl retry exponential backoff script will refine the exit conditions later.

max_attempts=5
attempt=0
until [ $attempt -ge $max_attempts ]; do
  attempt=$((attempt+1))
  http_code=$(curl -sS -o response.json -w '%{http_code}' \
    -H "Authorization: Bearer $API_KEY" \
    -d "$DATA" "$ENDPOINT")
  if [ "$http_code" = "200" ]; then
    break
  fi
  echo "Attempt $attempt failed with $http_code" >&2
done

This retries everything, which is wrong. We tighten the condition in Step 6.

Step 4: Implement exponential backoff

Sleep between attempts grows as a power of two. A base of 1 second is sane for LLM APIs; it respects typical rate limits without wasting minutes.

base_sleep=1
if [ "$http_code" != "200" ]; then
  sleep_time=$(( base_sleep * (2 ** (attempt-1)) ))
  echo "Sleeping ${sleep_time}s" >&2
  sleep "$sleep_time"
fi

At attempt 1, sleep 1s; attempt 2, 2s; attempt 3, 4s; attempt 4, 8s. Worst-case wait before the fifth try is 15s.

Step 5: Add jitter

Pure exponential backoff synchronizes retries across many clients and creates a thundering herd. Add a random fraction of a second using $RANDOM:

jitter=$(( RANDOM % 1000 ))  # milliseconds
sleep "$sleep_time.$((jitter))"

Bash sleep on Linux accepts fractional seconds. On macOS, install coreutils for gsleep, or compute with bc:

sleep "$(bc <<<"$sleep_time+$jitter/1000")"

Step 6: Retry only on transient conditions

Not every non-200 should be retried. 400, 401, 403 are permanent client errors. Retry on:

• HTTP 429 (rate limit)
• 500, 502, 503, 504 (server errors)
• curl exit codes 7 (connection refused), 28 (timeout), 35 (SSL handshake), 52 (empty reply)

Capture curl’s exit code separately:

curl -sS -o response.json -w '%{http_code}' ... 
curl_exit=$?

Then decide:

case "$http_code" in
  200) retry=false; break ;;
  429|500|502|503|504) retry=true ;;
  *)
    if [ $curl_exit -ne 0 ]; then retry=true; else retry=false; fi ;;
esac

if ! $retry; then
  echo "Fatal: HTTP $http_code / curl $curl_exit" >&2
  return 1
fi

This prevents burning attempts on auth failures.

Step 7: Enforce timeouts and a total budget

Never let a single curl hang your pipeline. Set --connect-timeout 10 --max-time 60. Also cap total script runtime so a degraded dependency can’t stall a job for hours:

start=$(date +%s)
# inside loop, before sleep:
now=$(date +%s)
if [ $((now - start)) -gt 300 ]; then
  echo "Total timeout exceeded" >&2
  return 2
fi

Step 8: Package as a reusable function

Put it all together in a function that takes a URL, a JSON payload, and an output file. This is the complete curl retry exponential backoff script:

llm_post() {
  local url="$1" data="$2" out="${3:-response.json}"
  local max_attempts=5 base_sleep=1 attempt=0 start=$(date +%s)
  until [ $attempt -ge $max_attempts ]; do
    attempt=$((attempt+1))
    http_code=$(curl -sS --connect-timeout 10 --max-time 60 \
      -o "$out" -w '%{http_code}' \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $API_KEY" \
      -d "$data" "$url")
    curl_exit=$?
    if [ "$http_code" = "200" ]; then return 0; fi
    case "$http_code" in
      429|500|502|503|504) retry=true ;;
      *) [ $curl_exit -ne 0 ] && retry=true || retry=false ;;
    esac
    if ! $retry; then
      echo "Fatal HTTP $http_code (curl $curl_exit)" >&2; return 1
    fi
    sleep_time=$(( base_sleep * (2 ** (attempt-1)) ))
    jitter=$(( RANDOM % 1000 ))
    sleep "$sleep_time.$((jitter))"
    now=$(date +%s)
    if [ $((now - start)) -gt 300 ]; then
      echo "Budget exceeded" >&2; return 2
    fi
  done
  echo "Gave up after $max_attempts attempts" >&2
  return 3
}

Call it from a script:

API_KEY="$OPENAI_API_KEY" \
llm_post "https://api.example.com/v1/chat/completions" \
  '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  out.json
echo "exit: $?"

Step 9: Verify the script works

You must confirm retries actually fire and that permanent errors bail fast. Three checks:

1. Force a transport error. Point the function at http://127.0.0.1:9/ (discard port). curl exits 7 immediately. Run with bash -x:

   bash -x yourscript.sh 2>&1 | grep -E 'sleep|http_code|retry'

   You should see escalating sleep values and the loop continuing until the attempt cap.

2. Simulate a 503 then 200. Use Python’s http.server with a custom handler, or a tiny Flask app:

   from flask import Flask
   app = Flask(__name__)
   c = {"n": 0}
   @app.route("/chat", methods=["POST"])
   def chat():
       c["n"] += 1
       if c["n"] < 3:
           return ("", 503)
       return ('{"ok":true}', 200)
   app.run(port=8080)

   Point llm_post at http://127.0.0.1:8080/chat and confirm it returns 0 after two sleeps.

3. Confirm fatal path. Temporarily set API_KEY=invalid against a real endpoint. The function should print Fatal HTTP 401 and exit 1 without sleeping.

If all three pass, your curl retry exponential backoff script is behaving correctly.

Operational notes

• Idempotency: GETs and read-only LLM POSTs are safe to retry. Never retry state-changing endpoints unless you send an idempotency key.
• Logging: Write attempt counts to stderr, keep the last response body on disk for post-mortems.
• Gateway fallback: If you call a gateway with automatic fallback when a provider is degraded (n4n.ai does this), transport-level retries still matter for local network blips, but you can lower max_attempts because the gateway already routes around bad upstreams.
• Per-token metering: When retrying, you only pay for successful requests; failed attempts against a metered gateway incur no token cost, but watch your rate limit headers on 429s to tune base_sleep.

That’s a complete, runnable curl retry exponential backoff script you can ship in a cron job or CI runner today.