Handling 429 rate limits in .NET LLM clients

When a C# service calls an LLM API at scale, encountering a dotnet 429 rate limit llm response is not a corner case—it is a routine failure mode. The HTTP 429 status signals the provider is throttling your requests, and a client that does not handle it will drop legitimate user prompts or cascade failures into downstream systems.

Step 1: Detect and inspect the 429 response

A 429 is just another HttpResponseMessage with StatusCode == HttpStatusCode.TooManyRequests. Before retrying, capture the Retry-After header and any provider-specific rate-limit headers so you can make informed backoff decisions. LLM providers often include x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset to tell you exactly how many tokens or requests are left in the window.

using var client = new HttpClient();
var response = await client.PostAsJsonAsync(
    "https://api.example.com/v1/chat/completions",
    new { model = "gpt-4o-mini", messages = new[] { new { role = "user", content = "hi" } } });

if (response.StatusCode == HttpStatusCode.TooManyRequests)
{
    var retryAfter = response.Headers.RetryAfter;
    response.Headers.TryGetValues("x-ratelimit-reset", out var resetVals);
    var reset = resetVals?.FirstOrDefault();
    // Buffer these values; they drive both backoff and alerting.
}

Do not assume the JSON error body is always present or parseable. The status code is the contract; the body is diagnostic. For streaming endpoints, a 429 typically arrives on the initial POST before any tokens stream, so the same detection works.

Step 2: Implement a basic retry loop with backoff

The simplest correct behavior is to retry with exponential backoff. Avoid fixed sleeps in production, but a minimal loop clarifies the mechanics and proves the path before you pull in a library:

async Task<string> CallWithRetry(Func<Task<HttpResponseMessage>> call, int maxAttempts = 5)
{
    var delayMs = 500;
    for (var attempt = 0; attempt < maxAttempts; attempt++)
    {
        var resp = await call();
        if (resp.IsSuccessStatusCode) return await resp.Content.ReadAsStringAsync();
        if (resp.StatusCode == HttpStatusCode.TooManyRequests)
        {
            await Task.Delay(delayMs);
            delayMs = Math.Min(delayMs * 2, 30_000); // cap at 30s
            continue;
        }
        resp.EnsureSuccessStatusCode();
    }
    throw new HttpRequestException("Exhausted retries on 429");
}

This handles transient throttling but ignores server-provided wait hints. In a dotnet 429 rate limit llm integration, ignoring Retry-After wastes time and may extend the outage because you retry before the quota window resets.

Step 3: Honor the Retry-After header

The Retry-After header can be a delta-seconds integer or an HTTP date. Prefer the explicit signal from the server over your local guess:

static async Task WaitIfThrottled(HttpResponseMessage resp)
{
    if (resp.StatusCode != HttpStatusCode.TooManyRequests) return;
    var ra = resp.Headers.RetryAfter;
    if (ra?.Delta is not null)
        await Task.Delay(ra.Delta.Value);
    else if (ra?.Date is not null)
    {
        var wait = ra.Date.Value - DateTimeOffset.UtcNow;
        if (wait > TimeSpan.Zero) await Task.Delay(wait);
    }
    else
        await Task.Delay(1000);
}

Wire this into the loop from Step 2: replace the fixed Task.Delay(delayMs) with await WaitIfThrottled(resp), but keep the capped exponential as a fallback when the header is absent. Some gateways omit Retry-After under burst limits; your client must survive that.

Step 4: Use Polly for declarative resilience

Hand-rolled loops get messy once you add circuit breaking, logging, and fallback. Polly is the standard .NET resilience library. Install via dotnet add package Polly and register the policy with IHttpClientFactory in ASP.NET Core:

using Polly;
using Polly.Retry;
using Polly.CircuitBreaker;

var retryPolicy = Policy
    .HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.TooManyRequests)
    .WaitAndRetryAsync(
        retryCount: 5,
        sleepDurationProvider: (attempt, ctx) =>
        {
            if (ctx.TryGetValue("retryAfter", out var ra) && ra is TimeSpan ts)
                return ts;
            return TimeSpan.FromMilliseconds(Math.Min(500 * Math.Pow(2, attempt), 30_000));
        },
        onRetryAsync: (outcome, _, attempt, ctx) =>
        {
            var ra = outcome.Result?.Headers.RetryAfter;
            if (ra?.Delta is not null) ctx["retryAfter"] = ra.Delta.Value;
            Console.WriteLine($"429 on attempt {attempt}; backing off");
            return Task.CompletedTask;
        });

var circuit = Policy
    .HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.TooManyRequests)
    .CircuitBreakerAsync(10, TimeSpan.FromSeconds(30));

var wrapped = Policy.WrapAsync(retryPolicy, circuit);

Execute your HTTP call inside await wrapped.ExecuteAsync(() => httpClient.SendAsync(req)). The circuit breaker stops calling a dead provider after consecutive 429s, which protects your thread pool from a thundering herd when the quota is exhausted for the minute.

Step 5: Add fallback to alternate models or providers

Retries alone do not solve sustained rate limits on a single model or provider. After exhausting retries, switch to a secondary model or a gateway that routes around degraded providers. An inference gateway such as n4n.ai can automate fallback when a provider is rate-limited or degraded, exposing one OpenAI-compatible endpoint that addresses 240+ models and honoring client routing directives. In your own code, implement fallback explicitly so you control cost and latency:

async Task<string> CallWithFallback(string prompt, string primaryModel, string secondaryModel)
{
    try
    {
        return await wrapped.ExecuteAsync(() => ChatCall(prompt, primaryModel));
    }
    catch (BrokenCircuitException)
    {
        // primary circuit open; go straight to secondary
        return await ChatCall(prompt, secondaryModel);
    }
    catch (Exception)
    {
        return await ChatCall(prompt, secondaryModel);
    }
}

Task<string> ChatCall(string prompt, string model) =>
    httpClient.PostAsJsonAsync("https://api.example.com/v1/chat/completions",
        new { model, messages = new[] { new { role = "user", content = prompt } } })
        .ContinueWith(t => t.Result.Content.ReadAsStringAsync().Result);

Keep the secondary call simple—same request shape, different model field or base URL. If you use a gateway, set a routing hint header instead of branching in client code. Be aware that fallback changes token cost and output format; validate the secondary response schema before returning it to callers.

Step 6: Track rate-limit metrics and break the circuit

A dotnet 429 rate limit llm event is a signal about your provisioned quota, not just a transient error. Emit counters with System.Diagnostics.Metrics or OpenTelemetry so ops can alert on quota saturation:

var meter = new System.Diagnostics.Metrics.Meter("llm.client");
var counter = meter.CreateCounter<int>("llm_429_total", "count", "429 responses received");
// inside onRetryAsync: counter.Add(1, new KeyValuePair<string,object?>("model", modelId));

Combine this with the circuit breaker from Step 4. After 10 consecutive 429s, the breaker opens for 30 seconds and all calls fail fast, giving the provider window time to reset. This prevents a retry storm from getting your account flagged for abuse.

For per-token metering, capture x-ratelimit-remaining on success and log it alongside usage. If you route through a gateway that performs per-token usage metering, those headers are authoritative; your client should not recompute them.

Verify success

You cannot claim resilience without testing the path. Stand up a local stub that returns 429 with Retry-After: 1 for the first N requests, then 200. A minimal ASP.NET controller works:

int _hits;
app.MapPost("/v1/chat/completions", (HttpContext ctx) =>
{
    if (Interlocked.Increment(ref _hits) <= 5)
        return Results.StatusCode(429); // add Retry-After header in real stub
    return Results.Ok(new { choices = new[] { new { text = "ok" } } });
});

Point your Polly-wrapped client at this stub and run 50 concurrent tasks:

dotnet run --project MyLlmClient --url http://localhost:5000
# observe logs: 5 throttles per task, then success

Confirm:

• No unhandled exceptions escape CallWithFallback.
• Logs show 429 counts increasing then dropping to zero.
• Total latency per successful call stays under your SLA despite throttling.
• Fallback fires only after retries exhaust (or circuit opens).
• Retry-After delays match the header, not your local exponential guess.

If you can saturate the primary until it 429s and watch the secondary serve traffic, the integration is correct. The dotnet 429 rate limit llm handling is done when your service degrades gracefully instead of throwing on the first throttle.