Migrating OpenAI SDK retry logic to a framework's defaults

Most teams start with the raw OpenAI SDK and layer on custom retry logic — exponential backoff, jitter, respect for Retry-After headers, circuit breakers. That code accumulates, drifts from provider behavior, and becomes a maintenance burden. This guide walks through an openai sdk retry logic framework migration so you can delete that code and rely on battle-tested framework defaults instead.

The OpenAI Python SDK has included configurable retries since v1.0 via the max_retries parameter and a Retries configuration object. Frameworks like LangChain, LlamaIndex, and Instructor expose their own retry abstractions that wrap the SDK. The migration is straightforward: identify your current behavior, map it to the framework’s knobs, verify equivalence, then remove the custom layer.

Step 1: Audit your current retry behavior

Before changing anything, capture what your custom code actually does. Search your codebase for openai.AsyncOpenAI, openai.OpenAI, or any wrapper class. Look for:

• max_retries passed to the client constructor
• Custom httpx.AsyncClient with HTTPTransport(retries=...)
• Tenacity or backoff decorators on completion calls
• Manual handling of openai.RateLimitError, openai.APIConnectionError, openai.InternalServerError
• Logic that reads Retry-After headers and sleeps
• Circuit-breaker state (failure counts, half-open probes)

Write down the effective policy: which status codes trigger retries, max attempts, base delay, max delay, jitter factor, and whether retries are idempotent-safe (GET/POST with idempotency keys).

# Example: what you might find today
import openai
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential_jitter

custom_http = httpx.AsyncClient(
    transport=httpx.AsyncHTTPTransport(retries=3),
    timeout=30.0,
)

client = openai.AsyncOpenAI(
    http_client=custom_http,
    max_retries=0,  # we handle retries ourselves
)

@retry(
    wait=wait_exponential_jitter(initial=1, max=30),
    stop=stop_after_attempt(5),
    retry=(
        retry_if_exception_type(openai.RateLimitError) |
        retry_if_exception_type(openai.APIConnectionError) |
        retry_if_exception_type(openai.InternalServerError)
    ),
)
async def chat_with_retry(messages, **kwargs):
    return await client.chat.completions.create(messages=messages, **kwargs)

Step 2: Choose the framework and locate its retry surface

Each framework exposes retries differently. The three most common integration points:

Open the framework source or docs for the exact parameter names. For LangChain 0.2+, the ChatOpenAI class accepts max_retries (int) and request_timeout (float or tuple). It internally constructs an openai.AsyncOpenAI client with those values and wraps invocation in a RunnableRetry if you call .with_retry().

# LangChain 0.2+ — canonical retry surface
from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnableRetry

llm = ChatOpenAI(
    model="gpt-4o-mini",
    max_retries=3,           # maps to SDK max_retries
    request_timeout=30.0,    # maps to SDK timeout
)

# Optional: add a retry wrapper with custom policy
llm_with_retry = llm.with_retry(
    stop_after_attempt=5,
    wait_exponential_jitter=True,
    retry_exceptions=(
        openai.RateLimitError,
        openai.APIConnectionError,
        openai.InternalServerError,
    ),
)

Step 3: Map your policy to framework defaults

Translate each knob from your audit into the framework’s vocabulary. The SDK’s max_retries counts retries, not total attempts — max_retries=3 means up to 4 total tries. Frameworks usually follow the same convention, but verify.

If your policy includes non-standard behavior — e.g., retry on 400 for specific error codes, or a custom circuit breaker — you may need a thin wrapper on top of the framework rather than replacing it entirely. But 90% of cases map cleanly.

# Mapping example: custom tenacity → LangChain with_retry
from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnableRetry
import openai

llm = ChatOpenAI(
    model="gpt-4o",
    max_retries=0,  # we'll control via RunnableRetry
    request_timeout=60.0,
)

# Equivalent to tenacity: wait_exponential_jitter(initial=1, max=30), stop_after_attempt(5)
llm_retry = llm.with_retry(
    stop_after_attempt=5,
    wait_exponential_jitter=True,
    retry_exceptions=(
        openai.RateLimitError,
        openai.APIConnectionError,
        openai.InternalServerError,
    ),
)

Step 4: Swap the client in your dependency graph

Replace the raw client instantiation with the framework-configured one. If you use a DI container (FastAPI Depends, injector, python-dependency-injector), update the provider. If you pass clients explicitly, update call sites.

# Before: raw client provided to services
# deps.py
def get_openai_client() -> openai.AsyncOpenAI:
    return openai.AsyncOpenAI(
        api_key=settings.OPENAI_API_KEY,
        http_client=custom_http,
        max_retries=0,
    )

# After: framework LLM provided
# deps.py
from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnableRetry

def get_llm() -> RunnableRetry:
    base = ChatOpenAI(
        model=settings.OPENAI_MODEL,
        api_key=settings.OPENAI_API_KEY,
        max_retries=0,
        request_timeout=settings.OPENAI_TIMEOUT,
    )
    return base.with_retry(
        stop_after_attempt=settings.OPENAI_MAX_ATTEMPTS,
        wait_exponential_jitter=True,
        retry_exceptions=(
            openai.RateLimitError,
            openai.APIConnectionError,
            openai.InternalServerError,
        ),
    )

Update type hints and any code that accessed client.chat.completions.create directly. Framework LLMs expose .ainvoke(), .abatch(), .astream() — use those.

# Service layer before
async def summarize(text: str, client: openai.AsyncOpenAI) -> str:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Summarize: {text}"}],
    )
    return resp.choices[0].message.content

# Service layer after
async def summarize(text: str, llm: RunnableRetry) -> str:
    msg = await llm.ainvoke([{"role": "user", "content": f"Summarize: {text}"}])
    return msg.content

Step 5: Add observability to verify equivalence

You need proof that the new retry behavior matches the old. Instrument three metrics:

1. Attempt count per request — histogram of total tries (1 = no retry, 2 = one retry, etc.)
2. Retry latency — time spent in backoff per request
3. Final outcome — success, rate-limited exhaustion, other error

The SDK emits openai.APIStatusError with response.headers.get("Retry-After"). Frameworks surface the same exception types. Wrap the LLM call to record metrics.

# observability.py
import time
from functools import wraps
from prometheus_client import Histogram, Counter

RETRY_ATTEMPTS = Histogram("llm_retry_attempts", "Total attempts per request", buckets=[1,2,3,4,5,6,7,8,9,10])
RETRY_LATENCY = Histogram("llm_retry_latency_seconds", "Time spent retrying")
RETRY_OUTCOME = Counter("llm_retry_outcome_total", "Final outcome", ["result"])

def observe_retries(func):
    @wraps(func)
    async def wrapper(*args, **kwargs):
        start = time.perf_counter()
        attempts = 0
        while True:
            attempts += 1
            try:
                result = await func(*args, **kwargs)
                RETRY_ATTEMPTS.observe(attempts)
                RETRY_LATENCY.observe(time.perf_counter() - start)
                RETRY_OUTCOME.labels(result="success").inc()
                return result
            except openai.RateLimitError as e:
                if attempts >= 5:  # matches stop_after_attempt
                    RETRY_ATTEMPTS.observe(attempts)
                    RETRY_LATENCY.observe(time.perf_counter() - start)
                    RETRY_OUTCOME.labels(result="rate_limited").inc()
                    raise
                # brief sleep to let metrics flush; framework handles real backoff
                await asyncio.sleep(0.01)
            except (openai.APIConnectionError, openai.InternalServerError) as e:
                if attempts >= 5:
                    RETRY_ATTEMPTS.observe(attempts)
                    RETRY_LATENCY.observe(time.perf_counter() - start)
                    RETRY_OUTCOME.labels(result="error").inc()
                    raise
                await asyncio.sleep(0.01)
            except Exception as e:
                RETRY_ATTEMPTS.observe(attempts)
                RETRY_LATENCY.observe(time.perf_counter() - start)
                RETRY_OUTCOME.labels(result="error").inc()
                raise
    return wrapper

Apply @observe_retries to your service functions during the migration window. Compare histograms before and after cutover.

Step 6: Run shadow traffic or canary

Don’t flip 100% at once. Route a fraction of traffic to the new code path.

Option A: Feature flag per request

# router.py
import random
from fastapi import Depends

async def get_llm(use_new: bool = False):
    if use_new or random.random() < 0.1:  # 10% canary
        return get_new_llm()
    return get_legacy_client()

Option B: Shadow mode — call both, log discrepancies, return legacy result.

async def summarize_shadow(text: str, legacy_client, new_llm) -> str:
    legacy_task = asyncio.create_task(summarize_legacy(text, legacy_client))
    new_task = asyncio.create_task(summarize_new(text, new_llm))
    legacy_result = await legacy_task
    try:
        new_result = await asyncio.wait_for(new_task, timeout=5.0)
        # Compare: latency, token usage, output similarity
        log_comparison(legacy_result, new_result)
    except Exception as e:
        log_shadow_failure(e)
    return legacy_result

Run the canary for at least one full traffic cycle (including peak). Verify:

• Attempt-count histogram matches (median 1.0, p99 ≤ 3)
• No increase in rate_limited outcomes
• Latency distribution unchanged or improved
• Output quality identical (spot-check 50+ samples)

Step 7: Delete the custom retry code

Once metrics confirm parity, remove:

• Custom httpx.AsyncClient with transport retries
• Tenacity/backoff decorators
• Manual Retry-After parsing
• Circuit-breaker state machines
• Any wrapper classes that existed solely for retries

Keep only the framework LLM initialization and the observability wrapper (or migrate observability to your standard middleware).

# Example cleanup
git rm src/llm/retry_wrapper.py
git rm src/llm/custom_http_client.py
# Edit deps.py, services/* to use framework LLM only

Run the test suite. Pay attention to tests that mocked the old client — they’ll need updating to mock the framework’s ainvoke/abatch/astream.

Step 8: Document the policy in one place

Add a RETRY_POLICY.md or inline config doc so future maintainers know the effective behavior without reading framework source.

# LLM Retry Policy (via LangChain ChatOpenAI + RunnableRetry)

- Max total attempts: 5 (max_retries=4 in SDK, stop_after_attempt=5 in RunnableRetry)
- Backoff: exponential with jitter, initial 1s, max 30s
- Retried exceptions: RateLimitError (429), APIConnectionError (network), InternalServerError (500)
- Retry-After header: honored automatically by SDK
- Idempotency: caller must provide Idempotency-Key header for non-GET requests
- Timeout: 60s total per request (connect + read)

Verification checklist

Before closing the PR, confirm each item:

• Metrics show identical attempt-count histogram (Kolmogorov-Smirnov test p > 0.05)
• No regression in error-rate dashboard for 24h post-deploy
• Latency p50/p95/p99 within 5% of baseline
• All integration tests pass with new client
• Custom retry files deleted, no dead imports remain
• RETRY_POLICY.md committed and linked in README
• Team notified; runbook updated for on-call

Common pitfalls

Double retrying — If you leave max_retries=3 on ChatOpenAI and add with_retry(stop_after_attempt=5), you get up to 20 attempts (4 × 5). Set one layer to zero.

Streaming retries — astream() retries are tricky. Most frameworks only retry the initial request, not mid-stream failures. If you need stream resilience, implement a higher-level retry that re-invokes the stream from the start.

Provider-specific headers — Some gateways (including n4n.ai) forward provider Retry-After and cache-control hints. The SDK honors Retry-After automatically when max_retries > 0. If you strip the SDK retry layer entirely, you lose that. Keep max_retries ≥ 1 on the client.

Timeout vs. retry budget — A 60s timeout with 5 attempts and 30s max backoff can exceed the timeout. Set request_timeout to cover the full retry budget: timeout ≥ max_attempts × (base_delay + max_backoff).

The migration is complete when your codebase has zero custom retry logic for OpenAI calls, the framework defaults match your documented policy, and observability proves it in production. You’ve traded maintenance burden for a single source of truth — and the next SDK update handles new error codes without you lifting a finger.