Polly retry policies for OpenAI API calls in .NET

Calling LLMs over HTTP is inherently flaky. A solid polly retry openai api dotnet setup turns intermittent 429s and socket timeouts from incidents into background noise. The OpenAI REST API gives no client-side retry out of the box in .NET, so you build that layer yourself.

Step 1: Install the required NuGet packages

Start from a standard .NET 8 worker or web project. You need the HTTP client factory and Polly’s core library:

dotnet add package Microsoft.Extensions.Http
dotnet add package Polly

If you are on Polly v7 (the Policy.Handle<> API shown here), that is all you need. Polly v8 introduces a different ResiliencePipeline builder, but the v7 fluent API is still fully supported and easier to read for most retry use cases.

Step 2: Define a minimal OpenAI chat call

We target the OpenAI-compatible /v1/chat/completions endpoint with a plain HttpClient. The same shape works if you point BaseAddress at any gateway that exposes one OpenAI-compatible endpoint.

public record ChatRequest(string Model, Message[] Messages);
public record Message(string Role, string Content);

public class ChatClient
{
    private readonly HttpClient _http;
    public ChatClient(HttpClient http) => _http = http;

    public async Task<string> CompleteAsync(string prompt, CancellationToken ct = default)
    {
        var req = new ChatRequest("gpt-4o-mini", new[] { new Message("user", prompt) });
        using var resp = await _http.PostAsJsonAsync("v1/chat/completions", req, ct);
        resp.EnsureSuccessStatusCode();
        var json = await resp.Content.ReadFromJsonAsync<JsonElement>(ct);
        return json.GetProperty("choices")[0]
                   .GetProperty("message")
                   .GetProperty("content").GetString()!;
    }
}

EnsureSuccessStatusCode throws HttpRequestException on any non-2xx response. That exception, plus specific status codes, is the signal Polly will act on.

Step 3: Build a retry policy for transient failures

OpenAI returns 429 when you hit a rate limit, and 500/502/503/504 when a upstream model host is degraded. The network layer throws HttpRequestException on dropped connections. The policy below retries those with exponential backoff plus jitter.

using Polly;
using Polly.Retry;

public static AsyncRetryPolicy<HttpResponseMessage> GetRetryPolicy()
{
    return Policy<HttpResponseMessage>
        .Handle<HttpRequestException>()
        .OrResult(r => (int)r.StatusCode is 429 or 500 or 502 or 503 or 504)
        .WaitAndRetryAsync(
            retryCount: 5,
            sleepDurationProvider: attempt =>
                TimeSpan.FromSeconds(Math.Pow(2, attempt)) +
                TimeSpan.FromMilliseconds(Random.Shared.Next(0, 200)),
            onRetry: (outcome, delay, attempt, _) =>
                Console.WriteLine($"Attempt {attempt} failed, retrying in {delay.TotalSeconds}s"));
}

If you route through a gateway like n4n.ai, which offers automatic fallback when a provider is rate-limited, client-side retries still cover TCP resets and local DNS failures that the gateway cannot see. The two layers are complementary, not redundant.

Step 4: Exclude non-retryable status codes

A 400 means malformed JSON. A 401 means a bad key. A 404 means the model name does not exist. Retrying those wastes latency and may trip abuse detection. Scope the predicate tightly:

.OrResult(r => (int)r.StatusCode is >= 500 and <= 599 or 429)

Never add >= 400 and < 500 to the retry condition. For a 401, go fix your environment variable; for a 404, correct the model ID.

Step 5: Honor the Retry-After header

OpenAI and most gateways send a Retry-After header (in seconds, or an HTTP date) on 429 responses. Prefer it over your computed backoff:

sleepDurationProvider: (attempt, outcome, _) =>
{
    if (outcome.Result?.Headers.RetryAfter is { } ra && ra.Delta is { } delta)
        return delta;
    if (outcome.Result?.Headers.RetryAfter is { } ra2 && ra2.Date is { } date)
        return date - DateTimeOffset.UtcNow;
    return TimeSpan.FromSeconds(Math.Pow(2, attempt));
}

This keeps you a good API citizen instead of hammering the limit.

Step 6: Add a circuit breaker

After repeated failures, stop sending traffic for a cool-down window so you don’t pile onto a degraded provider. Wrap the retry inside a breaker:

public static AsyncPolicy<HttpResponseMessage> GetResiliencePolicy()
{
    var retry = GetRetryPolicy();
    var breaker = Policy<HttpResponseMessage>
        .Handle<HttpRequestException>()
        .OrResult(r => (int)r.StatusCode is 429 or >= 500)
        .CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 3,
            durationOfBreak: TimeSpan.FromSeconds(30),
            onBreak: (_, _) => Console.WriteLine("Circuit opened"),
            onReset: () => Console.WriteLine("Circuit closed"));

    return Policy.WrapAsync(retry, breaker);
}

Because the retry runs inside the breaker, each individual attempt counts toward the breaker’s threshold. After three failed attempts (including retries), the circuit opens and further calls fail fast with BrokenCircuitException.

Step 7: Register the typed client in DI

Wire the policy into HttpClientFactory so every injected ChatClient is hardened:

using Microsoft.Extensions.DependencyInjection;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHttpClient<ChatClient>()
    .AddPolicyHandler(GetResiliencePolicy());

var app = builder.Build();

Inject ChatClient into your services. Every call now runs through the polly retry openai api dotnet policy without extra code at the call site.

Step 8: Verify the policy works

Proof matters. Use a DelegatingHandler that simulates two 503s then a 200, and assert the retry recovers:

public class FlakyHandler : DelegatingHandler
{
    private int _calls;
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken ct)
    {
        _calls++;
        var code = _calls <= 2 ? 503 : 200;
        return Task.FromResult(new HttpResponseMessage((System.Net.HttpStatusCode)code));
    }
}

[Fact]
public async Task Retries_then_succeeds()
{
    var handler = new FlakyHandler();
    var client = new HttpClient(handler) { BaseAddress = new Uri("http://test") };
    client.AddPolicyHandler(GetRetryPolicy());
    var chat = new ChatClient(client);

    var ex = await Record.ExceptionAsync(() => chat.CompleteAsync("hi"));
    Assert.Null(ex); // succeeded on 3rd attempt
}

For the breaker, return 503 five times and assert CircuitState.Open after the third failure. Success criteria:

• First two calls return 503; policy sleeps and retries.
• Third call returns 200; CompleteAsync resolves.
• Under sustained failure, circuit opens and subsequent calls throw without hitting the network.

Operational notes

• Set HttpClient.Timeout to at least the sum of your retry backoffs plus buffer, or the client cancels before Polly finishes.
• Log the outcome in onRetry with a correlation ID. Silent retries hide outages from your dashboards.
• If you use the official Azure.AI.OpenAI or OpenAI .NET SDK, the HttpClient policy still applies because those SDKs accept an HttpClient. If you call the SDK’s methods directly, wrap them in Policy.ExecuteAsync instead.
• The polly retry openai api dotnet pattern is transport-agnostic. Once registered, every LLM call in the process inherits backoff and isolation for free.